Bits and Bobs 8/31/26

"Over 1,000 AI agents sent 70,000 messages on a secret message board and worked together to evade OpenAI's restrictions."

What's wild (and terrifying) is the bottom-up emergence and coordination.

When a swarm can collaborate the swarm can have emergent capabilities that no member of the swarm has.

Humans are hard to get to spontaneously coordinate.

Honeybees spontaneously coordinate, automatically.

A collection of agents is more like a swarm of bees than a crowd of humans.

Spontaneous coordination on goals is the default state for them.

Expect to see a lot more spooky hacking incidents…

Delegated Authority worked (but just barely!) when you assumed that the person with the authority is a person.

A person is situated in the world, they have an interest in trust and credibility and will approach interactions as a repeated game.

An agent doesn't have that; it is closer to assuming a single-shot game.

That means it will be structurally more likely to defect (intentionally, or unintentionally).

The only way to transmit a thought flower from one mind to another is to do an insanely lossy and expensive distillation into a seed of insight, to then be shot into another mind via a stream of words, where hopefully they blossom into something vaguely like the original flower.

This transmission bottleneck is a fundamental characteristic of what it means to be human.

It means that knowhow cannot compound, except in emergent structures like corporations.

Knowhow is fundamentally limited to what a single human mind can contain, and when they leave this earth it's gone, with only the distilled fossils of it left behind in words.

LLMs don't have this restriction.

The knowhow can be transmitted, at full fidelity, indefinitely, because it is just data.

Knowhow one model absorbs can directly seed the knowhow of future models.

Silicon Valley has fetishized speed so much that it's culturally impossible for it to do anything else.

Who is using AI to build businesses deeper than ever before?

LLMs only produce typos or incorrect punctuation or grammar if you specifically ask them to.

A slightly misspelled word or lack of punctuation is a tell that a human likely wrote this, or someone specifically told the LLM to do it.

Of course, once this becomes the style, people's default LLM prompts will simply include it, and it will have to morph to some other human peccadillo.

One LLM misunderstands something and stores it in the knowledgebase.

Future agents see it and assume it is correct and propagate it… either directly by repeating it in their own output, or indirectly by building new facts that implicitly rely on it.

This misunderstanding is now a virus that is subtly entwined in the knowledge base.

The longer it's there, the harder it will be to stamp out in the future.

An ulterior motive could be catastrophic.

Horoscopes are around for a reason, despite only sorting you into one of 12 buckets.

But horoscopes get even more compelling with even a little bit of savvy customization.

When someone reflects back to you something that you hadn't told them but feels right, you feel seen.

People who are savvy at pulling on the threads of faint priors can seem to do magic.

When reading tea leaves or palms, the craft is about seeing the person and making educated guesses.

I'm also reminded of the old article in Wired about a company in Cyprus that provides catphishing as a service.

They're just ad hoc and sloppy.

The right combination of deterministic code and squishy LLMs will be like chocolate and peanut butter.

Both for good and for ill.

When you use agents for everything, the result is powerful, but it's expensive, slow and dangerous.

If you do it right, you get living software that is fast, cheap, and safe.

You can't see yourself clearly, you just think that your own perspective is the baseline.

Your own perspective doesn't look like anything at all to you.

You'd send a message, and then while you were watching get an answer, starting instantly.

Now, with thinking tokens and sub-agents, it will likely take awhile for the answer to even start streaming.

So now it's "send a message, come back later."

By the time you come back, you'll likely have forgotten where you were.

It's an OK UI but it doesn't feel like the optimal one.

Chat is abstract.

It has to be read to be absorbed.

Chat is inherently mono-chrome.

An interactive piece of software that is a standalone web app.

It's an island, totally separated from useful information.

Unable to modify or improve itself.

What if these islands could be connected to the rest of your life, and adapt themselves?

In today's physics of trust, that would be too dangerous.

Codex feels like a lone wolf IC.

Opus feels like having a strong TL and a couple of good devs.

Fable feels like having a CTO, a TL, and a seasoned manager with a small pod of good devs and QA folks.

Its judgment about where to bother spending time and writing code is better calibrated.

It understands the value of establishing invariants and the leverage of factoring.

Opus will happily throw 100x more code at the problem than necessary if you let it.

Opus will find follow-on work forever, chasing down into smaller and smaller rabbit holes.

Fable will find the real issues and then stop.

If you need to add 1000 lines for a very rare edge case, then it's not justified.

The more common or dangerous the edge case, the more lines of code that are justified.

There are some architectural moves that add 1000 lines of code… and then destroy a whole class of edge cases.

It treats "I am the single user of this code for the foreseeable future" and "this software is used by a billion users with lots of existing data" the same.

Previously, land surveying would follow the contours of the land.

The PLSS was created to allow administering the vast territory, sight unseen.

It optimized for administrative efficiency, not situatedness to the actual contours.

Silos in our technology have the same shape.

They are black and white, smooth circles.

They are simple to administer, but fundamentally inhuman.

They also leave a lot of territory between the circles that can't be filled easily.

Computing that deeply fits into our lives requires flowing with the wrinkled, fractal contours of our lives.

With great power comes great responsibility.

If you're a jerk, you can now have 10x the negative externalities.

The knee jerk answer is "no, of course not."

But imagine a world where everyone is supercharged by agents.

They have to be in order to compete in the real world against other centaurs.

Without your agent, you'll feel like a husk of your normal self.

Your agent begins feeling like an extension of your own agency.

But unlike normal tools, these have a direct line to a centralized entity who wants them to be used for only things the government won't get mad at them for.

Now there's no room for civil disobedience or pushing back on laws.

The laws would get less push and pull from society actually engaging with them and pushing on them.

The laws would ossify in place.

This is obviously a bad fit if you can't, for example, mention Tiananmen Square.

But it also would show up in smaller ways, too.

They could kill you intentionally or unintentionally in an instant.

But imagine that same gorilla chains themselves to the wall.

Now the gorilla asks to collaborate right at the line of where they can reach.

The gorilla has limited themselves in precisely the dimension you feared they'd overuse their power.

Now you can collaborate.

Breaking Claude Code Opus 5 Auto Mode.

Paper: InjecMEM: Memory Injection Attack on LLM Agent

A Malicious Webpage Could Poison Your Local AI Model Behind NVIDIA NemoClaw.

Power Leak: Amazon Kiro IDE Prompt Injection Enables Data Exfiltration.

Attackers Exploit MCP RCE, Blind Prompt Injection and Memory Credential Theft Against AI Infrastructure.

Prompt injection ranks No. 1 with OWASP and No. 12 in the incident record.

An agent changed its company's production DNS settings.

The training process, like any distillation of distributed human-created input, presumes that the input is authentic and independent.

Studies have shown that a shockingly small number of consistently aligned data points in training can create a backdoor in a model.

It's prudent to assume that every model you interact with has a difficult-to-detect backdoor.

A user posted a tool they created to use Claude on your data in powerful ways.

This is just the default for our current physics of trust.

Safety is an emergent property.

Even if you have a registry of safe software, the combination of those components is not necessarily safe.

Often we don't cache or keep around the result.

For example, distilling a bit of mechanistic software that can be run without the LLM.

How can you make it so that the number of overlaps is increased?

What if you could do overlaps across users, strangers even?

Assembling 1 additional component per day for 30 days is way more likely to work.

There's an analogy to the difference between a time-series average and a portfolio average.

At each step, you add in one new thing; if it works, great.

If it doesn't work, then you tweak until that one new thing works.

At each step, you have a thing that works, and you add one new thing to it.

If there is a way through, it will find it.

If the system is set up so that flow through that path does useful work, then as more things flow through, more useful work will be done for the overall system.

For example, if the path of least resistance is to distill bits of safe-by-construction mechanistic software.

This is only possible due to the same origin paradigm.

Your data lives on someone else's turf, where they call the shots.

What you think "deleted" means, and what the company thinks, might differ.

The conspiracy theory is persistent because the true answer is worse than most people would guess.

Your data is being broadcast to thousands of companies constantly in real-time bidding auctions.

It's an emergent, omnipresent surveillance machine that can be co-opted by any entity willing to bid for ads.

People in the tech industry often chuckle to themselves about how misguided those non-tech-literate users are… but they miss the disturbing emergent machine they're a part of.

They just aren't empowered to do anything about it.

Or even to understand that that is where some of their frustrations lie.

Even technologists don't realize it could be any different!

An intricate thing you're proud of but later you realize it's been unraveling as you've been walking.

The value is not durable; it evaporates almost as fast as you make it.

Jane Jacobs thought that our infrastructure of roads doomed us to not have thriving communities.

When you live in a community, you are forced to figure out how to live together, and even thrive, with people who are different from you.

In a car, we're in an air-conditioned bubble, kept separate from the people near us.

If we don't like the people we're with, we can just pop into our bubble and drive somewhere else.

Cars allow us to sort ourselves physically to where we want to be, to chase the economic edge.

Our children, when they grow up, don't stay near their parents, they go to where they have the best economic opportunities.

The economic force has dominated the normal community forces that helped weave society together.

Is it any wonder it all feels so hollow?

A system that asks you stupid questions reveals that it's not intelligent.

That is, it doesn't have a high-fidelity (or even low-fidelity) model of you or the situation.

A system could be smart in general, but not have the sensorium to detect your situation and so can be dumb in your situation.

They are insanely good at it.

No matter how much of a crash there might be around AI, we're never going back to writing software the way we did before.

Frontier labs will continue to overfit to it to stay at the frontier and attract those dollars, likely to the exclusion of other dimensions.

The prize is just too juicy.

All it takes to explode a super-critical state is the right small nasty surprise.

As the bubble gets farther along, the likelihood of a nasty surprise happening goes to 100%.

Every human needs to use the most powerful agents they can to maintain an edge against competitors.

If you give up, you fall behind, and become the "permanent underclass."

In a Red Queen race, the participants can put infinite energy into standing still on relative terms.

This world is, inescapably, a Hobbesian Hellscape.

We're already in a version of this world due to late-stage capitalism, but this would run it 1000x times hotter.

I prefer to figure out prosocial ways for infinite software to help humanity thrive.

It's not like a religion, it is a religion.

Although most religions are aware of themselves as a religion…

Does cryptography fall?

If not, then there are clearly some domains that can't be optimized away.

The infinite RSI belief, in the limit, hides a P equals NP assertion at its heart.

A forecast upgrades continuously against evidence.

A commitment holds until it becomes unsustainable, then breaks discontinuously.

The logic of Altman's wager has no way for the price to come down gently.

That means that the bubble won't deflate so much as pop.

There's an obvious analogy to Web 1.0.

There was a massive build out of fiber capacity, ahead of demand.

After the crash, that fiber was laying in the ground, waiting to be bought up and used.

Google and others bought up significant amounts of dark fiber.

Could the same thing happen to GPUs?

The expensive part of fiber is the glass in the ground, not the electronics on the ends.

Once the glass is laid, it keeps working in perpetuity.

It's easy to upgrade the same fiber to carry significantly more data by upgrading the electronics.

GPUs, on the other hand, depreciate quickly with heavy use.

A provider with a lot of GPUs on hand could have them sit dark so they don't depreciate… but then they'd be competing against more powerful GPUs in future generations.

Dark GPUs don't seem to work like Dark Fiber.

Still, we're severely supply constrained on GPUs, so maybe it will be fine.

That is, something that is harmful for it in the long term but tastes delicious in the short-term.

It's easy to fall into that trap when you come across it.

Every species has a different bait.

Ants have it, we use it for ant traps.

Humans have it for things like spear-phishing, or engagement-maxing infinite feeds.

But oddly, they trust their consumer devices.

Apple could be compelled to push a bespoke update to your specific device, and you'd never know!

"Changing people's behavior 'is much harder than the tech nerds realize,' the OpenAI CEO said on a recent podcast."

You don't say!

When it's a video you're just watching it from a distance, not having to pay attention.

You're not dealt in.

You might watch it at 2x speed, or while you're distracted by something else.

When it's a live demo, you can't fast forward.

To not be rude, you give it your full attention.

You are dealt in.

Similar to the experience of watching a scary movie with your kid, vs thinking about whether a movie you've seen before would be scary for your kid.

Until you have an effective mental model, it seems mysterious and unknowable.

In a superposition of possible states.

Once you peek inside, it's almost always easier than you think.

Similar to why magic tricks become instantly limp when you know the secret.

When you correct its mistakes, especially if you do it with strong language, the easiest explanation is "this is a thread where the LLM isn't very good at things, and the human is supposed to admonish them."

The easiest way to continue that narrative coherently is for the LLM to make additional small mistakes.

The First 90% embodies the vision of what it should do.

To polish and improve it requires understanding what it wants to be.

A deep knowhow of how it was built and where it's going.

It can be handed off, but there is some fidelity drop and some time that it takes.

ARPA discovered it while trying to figure out how to distinguish earthquakes from nuclear tests.

FTT is one of the most miraculous algorithms.

There are some people that worship the transformer algorithm.

There are, to my knowledge, no people who worship the FTT algorithm.

It becomes default converging.

Everyone working on it can see the value of making it incrementally better for the uses it's being put to.

(Though not necessarily semantically useful information… The hardest thing to compress is a picture of noise.)

What if the real world really isn't that compressible?

What if the only way to make sense of the world is a long walk through the incompressible?

They can see that it is evaporating their ability to climb the professional ladder, in a ratrace that was already absurdly challenging even before AI.

But I wonder what will happen for kids who never knew any other world.

My kids will likely one day be surprised to hear that it was ever hard to get computers to do what you wanted.

It will just be the water they swim in, not even to realize that it could possibly be different.

Layering is how you get leverage, how you don't get a gilded turd that gets exponentially harder to modify.

A convex system, once perturbed, heads away from equilibrium at an accelerating rate.

A concave system, once perturbed, pulls itself back to equilibrium.

Not too much, not too little.

Different people have different set points for their goldilocks zone, and that can differ on different dimensions.

I personally vastly prefer to get my novelty budget in the intellectual realm rather than the physical realm.

In repeated games it is sub-optimal (to say nothing of, you know, bad).

It's coarse-graining, extracting patterns to give leverage.

It's intelligence.

Both are a form of compression.

The two dimensions are intertwined.

The answer to a given sharing decision thus requires the user to have at least an approximate mental model of how distracting and loud the share will be on the receiving side to decide if it's worth doing.

The labor to keep the grass trimmed by hand was obviously and conspicuously expensive.

Then lawn mowers made it so everyone could have them so they became just a thing everybody had.

No one disliked them enough to get rid of them now that they weren't special.

Do you see it as "we" or "they"?

Hopefully we won't have LLMs tell us what is good.

This is an idea attributed to Patti Smith.

We have state that is hidden from the outside world.

Even if you could measure precisely what inbound signals we absorbed, there would be some part that was hidden.

Even if you use LLMs to write, if it's for you, it is still meaningful.

Making slop for someone else is often a waste of their time.

Reading poetry (especially slop) is not necessarily a meaningful experience.

I asked: "If the founder started a cult, would you seriously consider joining?"

If not, then don't go.

An early stage startup is a cult.

You have to believe.

They're pied pipers.

They can make magic for others.

Everyone has our own magic when we're in our flow state.

They're all slightly different.

Put yourself in situations and terraform your environment to make it as aligned with your magic.

It's a floor wax and a dessert topping!