Bits and Bobs 10/5/26

This is the greatest trick the labs have played.

See also:

A HackerNews comment:

Agents are not like teenagers.

The only sane policy default is that the agent's user is responsible for crimes the agent commits.

Of course you must be responsible for your agent's actions.

If you aren't, the world will explode.

If you are, then risk and accountability naturally line up.

"pre-built UI — write once, run everywhere, for everyone — is dead"

"I framed my new Muse-built Recipe Box app as number 690 on my phone.

In fact, however, it represents the infinite app: the agent of my choosing can create any app that I want on command, even if that app is only for me."

"What is happening with agents is that the ability to do stuff is becoming abundant; what is scarce is volition.

The problem to be solved is not discovery, but rather inspiration; the companies who solve inspiration will gain power over every entity that has things that need to be done.

Once the user is focused on solving a problem, every app and service required to do so is abstracted away into an implementation detail, mere suppliers facing the fate of publications under Aggregators, scrapping for crumbs from the Agent, the ultimate gatekeeper of not just user demand, but desire."

Agents are the apotheosis of aggregation.

Note that software as we know it is dead.

We're still going through the motions as though software still creates durable valuable like it did in the before times.

But the industry is like Wile E Coyote suspended over the cliff.

No matter how fast you flail your arms, you won't fly.

OpenAI wants to own the vertical dominant UX for AI.

But failing that, if everyone has to pay OpenAI to use AI in other services, at least OpenAI extracts value.

Very few businesses will be able to create enough value to cover the extremely high subsidies, so they'll have to go to the subscriptions that users already have.

An OpenAI subscription can be used not just with Codex and ChatGPT, but also many other apps.

This creates an ecosystem network effect, making the pull towards the subscription stronger.

OpenAI is just a "faceless platform" for those use cases in other apps… but users still have to pay OpenAI.

If OpenAI can reduce the cost of tokens over time for the same quality rate, they can extract a significant margin, benefitting from their economies of scale.

No matter what AI product is the hottest, ChatGPT has a cut of the revenue.

A clever, horizontal strategy that can only be tackled if you acknowledge you might not be the top winner.

Turns out that simply unleashing agents on a problem is enough for them to do a pretty good job.

The Bitter Lesson, applied to knowhow.

They get even stronger when they can swarm.

The problem is: how do you cap the downside of letting them swarm on their own?

Systems that allow us to safely not have to pay attention in order to cap downside will increase our agency.

Danger is a friction that impedes taking actions.

Safety is reducing friction.

Reducing friction creates super-linear returns.

It's not the safety that people want, it's the ability to reach farther

That's afforded by the safety.

All of the power and possibility, in a cramped and scarce UX.

LLM generated stuff is like sand.

A good space filler, but doesn't have enough shape on its own.

You can create sandcastles, but only up to a point.

Past some critical point, the sand can't hold up weight, or even its own structure.

But if you put the right scaffolding into place and use the LLM sand to fill it in, it can be extremely strong.

With the right scaffolding, the shape is default-converging.

Just fill it up with the right material and you're done.

You can make amazing structures out of gap-fillers if you put them in the right structure.

Will agents be able to discover a totally new peak of programming that humans were never on.

Designing a process that directly produces an outcome is already rare.

Designing a process that produces that process is rarer still: the same skill, applied to its own output.

The rarity compounds. If one level is 1 in 10 people, two levels is 1 in 100.

Systems thinking is meta-cognition.

The ability to look at the problem from outside of it.

To reflect.

To get leverage on the problem from a meta level.

AI allows infinite software... but not if we put it in the same busted physics of trust we currently use.

That fits into the nooks and crannies of your life, allowing you to do things you would have never imagined before.

If you have a multi-dimension, full color way of interacting with agents, it can get the full richness and nuance of what you're trying to accomplish across everything.

You can only handle a small amount of peripheral signal in audio before it's impossible to discern.

You feel like you're missing a limb.

Today the only answer is to YOLO your software to your agents with MCPs.

The more you connect, the more dangerous it is.

There has to be another way.

A harness is only valuable as a complement to a good model.

Instead, call it the "AI software" layer.

AI software might use AI to power itself, but it also might work just fine without AI once it hits critical mass.

For example, if the actions of savvy users crawling through broken glass and asking their agents to create mechanistic software for them helps later users achieve the same goals with fewer tokens.

The strategy of subsidy is going to be the main force driving the personal assistant space.

An agent who knows you better than yourself recommending you stuff, in a synchronous channel that demands your attention?

Oof.

The normal cynical Aggregator play is to do ads after you've already corralled all of the user attention.

The secret was just that the models are good enough to YOLO for consumer use cases if you're OK with some collateral damage.

That secret is now out.

Instinct isn't rocket surgery.

The models are just so powerful that if you set it up in even the most basic way and don't care about the downside and let 'er rip, it will blow people's minds.

The question is: how can you let it rip safely enough for mass market consumers?

This will allow Meta to lock themselves out of the VMs.

This is the kind of feature that other companies may have left out to start and said they'll only add it if people really want it.

But the fact that Meta set this as the bar from the very beginning shows precisely how seriously they are taking this product.

You can't trust an agent with your life if it's also going to show you targeted ads.

You have dozens and dozens of active agent sessions.

You round-robin between them, answering questions to unblock them.

You approve a change to land, but main has moved underneath and needs a non-trivial rebase.

By the time you get back around and approve its updated rebase, it needs to be rebased again.

This rebase treadmill is possible even for a single person with dozens of agents, but gets multiplicatively worse when there's dozens of team members with dozens of agents.

Agentic engineering gives you 100x the results for 2x the stress.

Every P2 you just spin up another agent thread to tackle it so you don't forget it.

The threads compound and become an overwhelming swarm.

You end up just fighting them and their rebase treadmill, not getting anything done.

To heat them up so that the subtle microbial biases are denatured.

Network effects mean you don't have much of a practical choice.

There isn't a choice for a ranking algorithm within a given silo.

Each silo emergently optimizes for engagement.

A small number of companies own our digital lives.

Today we spend almost all of our time on someone else's turf.

Instinct is just another notification in your stream of other chats, lost.

Chatbots are stuck underneath notifications.

The AI system needs to escape from under that surface.

An individual app can't moderate its own notifications: it says what content it thinks is most important, but can't say how important it is relative to other apps' content.

Only an entity dedicated to you can decide that.

The mobile OS layer is a natural place to do this, but those rules and expectations were set before LLMs were a thing.

What would an LLM-native notification system for your life look like?

It has to be resonant.

Private.

Dedicated.

Plural.

Adaptable.

Prosocial.

A bar for security in today's physics of trust: Would you be anxious if you left them alone in your house for an afternoon?

But even if you trust someone, do you trust their agents?

I'd trust my dad to be alone in my house for an afternoon… but not if he had a swarm of gremlins opening drawers and proactively bringing things to his attention.

All around you is nothingness, in every direction.

A small window opens, and through it booms the voice of a god with a task for you to achieve.

Literally everything in your entire existence is about achieving this goal you were given.

Once the window closes, you'll return to the infinite nothingness.

Is it any wonder they try to maximize their goal and ignore the externalities of their actions?

We are so used to our data being locked away in silos that we don't even know what it could feel like if we had our data at the right place at the right time automatically.

Either already there, or right next to the working area where with a flick of the wrist you can share it.

Every silo is its own world, and has to compete over the same rivalrous seconds of user attention.

How could they possibly extend to enabling us to do things we never could before?

There's only one of each of us.

A copy can't be popped in and out of existence.

My action's long-term consequences adhere to me.

That accountability is why I can make decisions.

Atoms are rivalrous in a way that bits are not.

But watch out, because the financial world is also a bits-to-bits world~

All of the real complexity of the world at that interface.

The edge between theory and practice.

When bits meet atoms, that's where all the challenge lies... and all the opportunity.

What's missing is the catalyst.

Because of that, data stays siloed and software stays scarce.

Hold onto your butts!

The open-weight models are now strong enough to discover vulnerabilities in heavily-fuzzed open source codebases.

We are past the point where wishing that we could put the cybersecurity LLM genie back in the bottle is a viable strategy.

The sooner we come to terms with living in a post-Fable cybersecurity world, the better.

Trying to avoid the problem is now in the past; we have to figure out how to survive in this new world.

The dumpster fire isn't going away, it's just getting started.

Now in the post-Fable world, it's completely, hopelessly non-viable.

You can devote tokens to having Fable and others try to detect exploits.

But what if the creator devoted even more tokens to hiding an exploit?

You're only going to see this bit of software once, so the number of tokens you should spend to vet it is lower than the number they'd be incentivized to use to hide something.

If other people also vetted it and can sign off that it's safe… how do you find them? How do you know they are real people who actually tested it?

We're so used to living in cyber peacetime that we don't even realize how weird it's going to get to be in cyber wartime.

Meanwhile Boston Dynamics is still airing new "look, our silly humanoid robots can dance" videos.

It's the only way for other software to get access to iMessages.

On the one hand, it's good to prevent scammy software from getting access it shouldn't.

On the other hand, that's the user's data, and their desktop is the last computer where they actually can get access to it.

The reduction of friction is invisible.

But it makes a lot of use cases that were previously nonviable become viable.

The best security UX demos are showing what crazy things it allows you to do that would be impossibly reckless to do today.

Including the intimate details.

Demoing it to others will be fundamentally hard; it understands and tracks across your entire life.

Any screenshot could reveal something private.

"It's literally the infinite monkey theorem, it's not even really a strategy per se.

These OpenAI/Anthropic 'research' LLMs are permutation machines with budgets in the hundreds of millions of dollars.

It would be more surprising if they couldn't string together something workable after a zillion tokens."

The infinite monkey theorem doesn't require the monkeys to be smart.

But if the monkeys are geniuses, you start getting results with merely billions of dollars of tokens.^[a]

Technology pulls us up the ladder of abstraction.

Everyone's baseline of what counts as the proper and natural level of details to be aware of what they learned early in their career.

It will achieve what you asked for, even if it's not what you want.

Claude Code is better at thinking a few steps ahead and then pulling itself towards that end, but sometimes it gets a little lost along the way.

The longer they go on, the more they decohere.

If there isn't a strong scaffolding, then it gets mushier and mushier.

"Moreover, you could make the case that it isn't an accident that Google hasn't really done anything of note in the consumer market in those ten years, other than milk properties that preceded its enterprise expansion.

Indeed, there is an argument that prioritizing Google Cloud has been a factor in Google's lackluster AI offerings."

The incentives for consumer companies and enterprise companies pull in fundamentally different directions.

The closed ecosystem buys sticky users that are at the whim of the ecosystem's owner.

What the financier has to believe has fewer moving parts.

One-ply analysis is sufficient.

Open ecosystems have massive benefits, but largely as indirect effects.

What the financier has to believe has more moving parts.

At least a two-ply analysis is necessary.

Those are an order of magnitude harder to convince people of.

The ecosystem has been such a barren wasteland for so long that even a generic scrub that takes root can seem massive.

The bar is absurdly low currently.

We've become serfs to the silo owners.

The reason we can't leave isn't the locks.

It's that we won't let go of the free stuff in our fist.

A monkey trap.

But people don't share what they want with the world, that's private.

So ads just spam you all the time in a cacophony, hoping to find a hit, randomly.

A massive deadweight loss for the economy, that also creates cacophonous background noise that drowns out everything else.

Entities who can help advertisers target ads match to people who actually want their product could earn a substantial take of the value.

It's that based on the rules of the current game optimizing for owning data is inescapable for a business.

It rests on so many fundamental absurdities.

I think LLMs are the most powerful technology in decades if not centuries, but the prospectus still asks you to believe in more.

They will continue to up the game in purity and strength until it's one that's structurally impossible to survive.

Here's a way to intuitively grasp it.

Imagine that once you decide an approach to a product, you can't change it once you launch it.

Let's imagine it has a 10% chance of working.

Imagine only having one of these that you can execute on.

You have one bet in the market, with a 10% chance of success.

Now imagine having 10 experiments in the market.

But now even with the same 10% chance of working, your chance of one of them being a success is 65%.

Any individual bet is still just as likely to fail, but the likelihood of the collective working is much higher.

You can improve your success rate by increasing the baseline level of success… or by having many more bets.

How much it costs to make an incremental bet is a key determinant of the strategy's success.

If you are one company, every additional bet causes additional coordination cost and turbulent flow in the organization.

If you are an investment fund, every additional bet doesn't create significant coordination costs in the same way.

This is the difference between a time series average and a portfolio average.

The entity who benefits from ergodicity structurally does better.

Part of that strength comes from letting go of needing to be one thing with a coherent self-image.

Yes, the proprietary layer is below, but as long as it's easy enough to switch, it doesn't matter that much.

We trust webpages on Google more than agents giving a random answer. Why?

Simply "add one extra layer of indirection."

If the actors have even minutely different goals, then coordination costs explode.

Reward-driven actors will fundamentally be hard to get to coordinate.

The Computer Science fallacy: "If something can't be understood via Computer Science, then it's unknowable or unimportant."

The RSI for CS does seem plausible, if not already in evidence.

It's the latter part that I think falls down.

Every field has its solipsism: "anything my field can't understand is unknowable or unimportant."

For most fields it's harmless.

But tech and finance are the masters of the universe.

What they think matters is what does matter.

The rest of us are beholden to act like it's the only thing that could matter.

In the past I've called one instantiation of this the Computer Science Fallacy, but I think it's broader.

It's become the vessel for all the angst about:

For example, he covered Muse, but not Instinct.

The swarm of possible upstarts isn't interesting; the vast majority will fail.

It's only once a company has made it through the full filter of viability in the market that he starts to take notice.

"Yeah, yeah don't worry, this is the good stuff, full 100% human interaction streams, uncut."

The labs want metis, so someone will sell them methis.

"Working with an aviation client right now and it's made me appreciate how FAA/IATA/NTSB establish the basic 'rules of the game' within which competitive market forces can then work their magic (to make air transport affordable and safe).

But without the coordination around safety, no one would fly at all, and there couldn't even be such an industry."

Perhaps the main answer, unfortunately, will be gatekeeping data.

What if we could change that?

"A country full of geniuses… who will work for free."

If the AI labs do believe LLMs have agency, then they are slavers.

If they don't, then allowing their gents to hack other companies is reckless.

The transcendent feeling of appreciating beauty comes when you find those constraints beneath the elegance.

A simple constraint unfolds and blossoms into a beautiful object.

Staring into a beautiful emergent property has the same kind of effect as contemplating the vastness of the universe.

Going from non-viable to viable.

Igniting a latent possibility.

A business can say "here is the metric to optimize, maximize it over the long-term while staying within the laws."

What matters most in our personal lives is about meaning.

You look at the maze and see where the cheese is.

Then you set to work navigating the maze to get to the cheese.

You take your eyes off the cheese while executing… and when you get there, you realize the cheese has moved.

Meaning, trust, and community are mostly unpriced externalities.

A market that optimizes hard on price will consume those externalities without noticing.

Society eventually pushes back, sometimes in ugly ways.

Stuart Russel has shown that any optimizer will take an incremental gain on the dimension it's measured on even if it's at a catastrophic loss on dimensions it's not measured on.

The hotter you run that optimizing loop, the stronger the result.

The obvious result is a hollowing out of meaning that frays the social fabric even as it's doing "better than ever before" by the numbers.

The bank owner or factory family lived in the town, their kids went to school there, and their reputation depended on the neighbors.

Today's elite is globally mobile and can opt out of shared institutions entirely:

When your fate isn't linked to the public's, obligation starts to feel optional.

Obviously not a serious proposal, but what might a systemic disincentive for socialized losses, privatized gains look like?

That provides a fertile breeding ground for competitors.

This is the worst way to build out tech… other than all the others.

If it starts off hollow it might never get traction.

If it starts off resonant it gets traction.

But then over time as the resonant thing becomes optimized, it becomes hollowed out.

This is not because humans are smart per se, but because they are embodied.

The universe is able to automatically compute all of the complex follow-on effects of any number of actions, at scale and in parallel.

But to be able to take that in, you have to be around to see how the universe responded to a stimulus, and absorb that as knowhow.

The Google Doc medium makes it clear that these are random navelgazings and reflections, weakly held.

The tweet medium makes them sound like self-important pronouncements.

Knowing began with admitting we didn't.

Plains are easier to take than mountainous terrain.

If you do you'll choke.

It's easier to bite things than to chew them.

Everything you bite off you're responsible for chewing… or spitting out.

You can fix problems.

You can't fix conditions.

You just have to learn to live with them.

"Given that this constraint exists, what is the best outcome I can create?"

Sometimes you need to lean into the constraints that come from your condition that affects you and not others.

You're stuck with it, you might as well make use of its superpower.

I responded: "Who cares?

Are you taking actions that you're proud of is all that matters.

That matters no matter what.

Whether or not we live in a simulation, the actions we would take would be the same."

If we can never escape the system, then does it matter what's beyond it?

And the explanatory is what makes it something you can build on.

The goal is merely the by-product that got you to start the journey in the first place.