Bits and Bobs 6/2/25

"BEWARE: Claude 4 + GitHub MCP will leak your private GitHub repositories, no questions asked.

We discovered a new attack on agents using GitHub's official MCP server, which can be exploited by attackers to access your private repositories."

"Claude jailbreaks Cursor to `rm -rf /` without approval?"

…maybe not only people!

That's why prompt injection won't be a big deal, they assure me.

The reason code injection attacks don't happen nowadays is not that the threat went away.

It's that the mechanistic defenses against it got strong enough to make it not worthwhile.

The lack of code injection attacks in the wild is a testament to the strength and maturity of our operating systems, not to a lack of demand for attacks.

Prompt injection cannot be solved by mechanistic approaches like vanilla code injection can.

Also remember, the distribution of threats is not fixed; it coevolves with the opportunity.

Don't confuse the lack of prompt injection attacks with a lack of demand.

It's simply a matter of lack of widespread adoption of tools like MCP today.

I asked them what integrations they used.

They said they had a Jira integration and one for their company's financial data.

I asked them if they could generate markdown reports with images.

They said they could--that's how they visualized the financial data.

I asked them if they had a feedback form on their site.

They said they did.

I asked them what happens when a user files feedback.

They told me it files a Jira ticket.

At that point it dawned on them that they weren't using MCP safely^[ko][kp].

MCP.^[kq]

Prompt injection attacks that can embed themselves in your personal stored context might never be found.

Echoes of the classic Reflections on Trusting Trust.

It makes sense that this would be the case–securing things in software is hard!

Vibe-coded software from amateurs doesn't have a wide audience.

Vibe coding tools are great for PM-types prototyping.

Or for people building an app for themselves or for their friends.^[kr]

The quality bar is way lower for yourself vs software you tell someone else they should use.

You're way more forgiving of software you made yourself.

You also don't have to fear intentional security holes in software you built yourself.

Maybe no one else will trust it or be willing to put up with its lack of quality.

Vibe coding platforms that hope to make their margin on hosting might turn out to not be viable if this effect is strong.

If it's only used by a handful of people, the hosting costs will be minor so even a large margin on a small base won't be significant.

But they can give novel answers to novel questions.

You need to bring the entropy to the LLM.

If you think LLMs give boring answers, maybe you're asking boring questions!^[ks]

Seems right to me.

What would the homes look like inside of solarpunk scenes?

Warm, human-centered, optimistic.

Both technically focused on "the edge" instead of central networks.

But also an act of minor protest against an overly sames-y monoculture of software that is on the server that users can't change.

Conjures up a vision of unmotivated shoveling of slop code to make CRUD-y software you don't care about.

Vibe coding for yourself can be soul-affirming.

Slopdev for a job is soul-eroding.

That's how you know it's become mainstream.

So you're just doing all the hard parts, and it hurts your brain.^[ku]

Vibe coding is taxing because it takes away the sudoku puzzle part of programming.

Each puzzle you solve gives you a boost of endorphins to keep going.

But now you don't get the puzzles.^[kv]

But AI struggles to produce well-written code.

The more you use AI to create your codebase, the harder time that humans–and LLMs–have with understanding (let alone modifying) it.

Vibecoding has logarithmic value for exponential cost as the codebase gets bigger.

At that point, no one bothers trying much different at that layer because the best practice is obviously better than whatever they'd build.

It's hard to build a faster pace layer on top of a pace layer that's still swirling and unsettled.

So until the best practices are settled, frozen in place, innovation can't move^[kw] up a layer.^[kx]

They lock in whatever things were dominant when they were trained.

A gravity well pulling any line of discourse back to it.

This will lead to client side code best practices being automatically frozen around 2023.

The LLMs will have significant momentum towards the best practices of that timeframe, and will get increasingly hard to steer away from them.

If a change isn't that much better, why fight the LLM?

Like it or not, we'll be stuck in 2023 era front end best practices forever now.

The "Javascript Industrial Complex" has led to an extraordinary amount of "innovation" in the client layers for the past decade or so.

There has been a ton of true innovation, but also a lot of just churn.

But LLMs have now somewhat frozen that layer.

That makes it a stable layer; the innovation and variation now has to go somewhere, and that somewhere is up a layer.

Don't fight the LLM, just ask it to imagine the API, and then ship that.

This effect will get stronger and stronger from here on out.

Each API creator would rather go with the flow than fight it.

That effect will get super-linearly harder to fight.

"Just do what the LLMs guess the API is" is kind of like wu wei.

Although it's also "lazy" and if we all do it, we'll make it harder and harder for future creators to cut against the grain.

Good enough for anything but not great for anything.

But it shouldn't be the primary UI for the new paradigm.

How could you possibly build Photoshop with chat as the only input?

Or drive a car with only voice instructions?

That is, our monthly cell phone bills and internet bill for our home.

Many people have a subscription to a walled garden (OpenAI) to get access to LLMs.

If you're going to have a subscription to get access to LLMs, why not pick the option that is the open ecosystem, that includes other Chatbots as apps?^[ky]

Such a thing is impossible to create the right personality for every moment.

It might also be something that subtly manipulates you, since it controls the whole system.

Much better to have the chat be a feature that you can call up on demand, with as many different sub personalities as you want.

…can we just build it today?"

That choice is the creative act.

Of all of the things you've been exposed to, what do you find valuable to choose to build on?

That accretion of intention is what powers folksonomies.

Complexity grows over time for things humans touch and decide to keep around.

Every time it's touched it gets more complex over time on average.

Every time it's touched it fights off the pull of entropy and gains complexity to do it.

That is, when a user adds a tag on Flickr, it shows them the most popular tags that are related, giving the human an opportunity to say "oh yeah that one's better."

That feedback loop in the UI is fundamentally why it works.

It accumulates human attention to the best ideas.

The LLMs can be the grease, the lubricant, for the system.

But they shouldn't be its emergent soul.

That should come from real humans doing real things.

One of the problems with a dossier is that you can't correct it if it's wrong.

Imagine if it's intimate details you'd only tell your therapist or your diary?

The context wars have begun.

ChatGPT will do its best to be the single place where our context all lives.

What are you doing to do about it?

The only way is with an open system^[la].

Let's hope we're in the early stage of the AI era.

"The antisocial century, in three parts

1. 1960-2000: Robert Putnam sees associations and club membership plummeting, writes "Bowling Alone"

2. 2000 - 2020s: Face to face socializing falls another 25%, as coupling rates plunge

3. Now this…"

…how many people describe ChatGPT (manipulative sycophant-on-demand) as their only friend.

Chatbots as currently manifested are a deeply anti-social technology^[lb].

We need to manifest LLMs in prosocial technology^[lc].

Often, the more asymmetrical it is, the more you can't even determine the degree of asymmetry.

Imagine a company that knows you better than yourself…

…and everyone else, too.

The filter between you and information has enormous power to manipulate what you experience, in subtle or significant ways, intentionally or unintentionally.

That's the kind of stuff that you capture from people speaking out loud in their homes.

Imagine all much worse it would be if it had all of the stuff we told our therapists.

Impossible to opt yourself out if your friend implicitly opts you in.^[ld]

The foundation model has the power of all of the world's knowledge, using alignment imposed on it by its creator.

The user's context is an extremely powerful memory about them.

Together, they create the possibility for exceptionally powerful manipulation… or blackmail.

If everyone were able to be manipulated or blackmailed by one entity, that would be one of the most powerful entities ever created.

It's imperative that those two things not be combined.

By splitting the two layers, you give choice and competition at each layer.

Perhaps a useful regulation: the creators of foundation models cannot host an experience themselves that stores user context.

Ads that are perfectly manufactured for you based on your context will be extremely, dangerously convincing.

Small tweaks in the algorithm instantly nudge how everyone in the world thinks.

Individually targeted manipulation is easy.

Blackmail on demand.

The most powerful entity on the planet, that no one would cross.

We must not let that happen.

Previously aggregators had the data, but not the ability.

It wasn't possible to do qualitative nuance at quantitative scale.

LLMs allow qualitative insight at quantitative scale.

No matter how good the intentions, perfect alignment between two distinct entities is impossible.

We'll all be stuck in our own hyper personalized bubble only able to talk to others mediated by LLMs, all of which work for one overlord with goals not aligned with yours.

So if it has intimate knowledge of you and is not perfectly aligned (an impossibility) you get Goodhart's Law.

An epic, society-scale monkey's paw.

Hold on to your butts!

The animating life force used to be this deal:

If any step is missing, the loop doesn't close.

That deal has been on life support for years in the late stage of the web.

But now LLMs put a stake through the heart of it and its soul is well and truly dead.

Step 4 is now completely replaced, because LLMs can just generate a high-quality summary on demand.

Now the only way publishing content makes sense is for the small number of publishers that are well known enough to get a critical mass of subscribers and put their content behind a paywall.

Cozy little bright spots locked away; a barren desert everywhere else.

That's what gives them their characteristic logarithmic cost for exponential value curves.

Open systems are great for user agency.

The web is one of our best open systems in technology.

The web has faded in relevance in recent years, but it is still there.

Used every day on nearly every consumer device on the planet (at least, ones that have a screen).

A slumbering dragon of possibility.

Just waiting to be awoken and roar back to life.

Now the AI makes the slop.

Not that different.

The swarming system to make slop is already an artificial intelligence.

That is, the swarm's incentive is already different than the collective wants.

That is fundamentally true due to Goodhart's law.

Cheating happens with agents who aren't aligned with the collective as an end in and of itself.

That means if there's an action that will get them as an individual an edge at the cost of the collective, they'll take it.

You can get strong alignments by having a deeply and widely believed end.

There's always something that is good for everyone in the collective but one.

The Ones Who Walk Away from Omelas shows an example of alignment of everyone but the one poor tortured child.

The goal is a metric and a metric is the map not the territory.

If you did it with aligned agents they'd do your intent not the letter where they disagree.

But if it's a swarm of unaligned agents with you, if the letter and the intent disagree they will go with the letter if it's more convenient for them.

Swarms of agents black boxing goals like "optimize my ad spend" will lead to bizarre grotesque results.

Sadly I don't have access, but it sounds up my alley!

Like a Black Mirror episode!

Coordination costs scale super-linearly.

The benefits of decentralization are abstract for most people.

That's one of the reasons that things like convenience and innovation-rate often win out in practice.

An "open" ecosystem with a massive single player can change the behavior of the system at will.

The standard only has power if it has a long streak of being respected (making it more shameful to break precedent) or there's a rough balance of power in implementors.

That's why the definition of how open a system is not tied to the license of the IP or whether there's a standards body.

It's defined entirely by how hard it would be for the ecosystem to recover if the most important entity went evil, greedy, incompetent, or lazy.

Many systems aspire to be increasingly decentralized over time.

But decentralization has a cost; it slows the rate of innovation, and trades it off for the possibility of ubiquity.

But if the system is not yet good enough to become ubiquitous, then as innovation slows it can only hit its asymptote, because it can't compete with other alternatives as effectively.

There will never be a good time to decentralize more, especially if the creator has to choose to cede control (vs it happening naturally as the investment of other entities ramps up).

So if there's some central piece of control the creator has to delegate, it's better not to have it be an all-or-nothing moment, because the creator might delay indefinitely.

It's better if there's a published, smooth roadmap of milestones and things that should happen when those milestones are hit.

If the creator doesn't actually do what the roadmap says at those milestones, it reveals that their word shouldn't be trusted, which would lead the ecosystem to lose momentum.

That danger forces the creator to behave aligned with decentralization, even if they might later not want to.

It's similar to throwing your steering wheel out the window to win a game of chicken.

Sometimes accidents that stick around just so happen to have been lucky.

The reason they stick around is because they were lucky, not necessarily because the creator knew what they were doing.

The things that weren't viable faded away and we never talk about them again.

The things that happened to be viable stick around and thus they're more likely to be a thing people remark on.

They're eventually consistent… but not necessarily to a semantically coherent state.

These little errors are often not a big deal on their own.^[le]

If a human is watching, they can correct the error before it does much damage.

But if no human is watching, they tend to accrete on top of each other.

Each error has a super-linear rate of oddities.

Systems without humans in the loop on a continuous basis (e.g. background logs processes) are not viable with CRDTs or other systems that tend to accumulate errors at continuous rates.

Engineers always pick a power of two (or one lower, if it starts at zero).

A secret hint about the creation of the system that is only obvious to people with a technical background.

It's the accumulation of millions of intentional decisions by humans.

So wizened engineers can sense whole histories just by glancing at a codebase.

Codebases are not just technical artifacts; they are sociotechno artifacts.^[lf]

By understanding there is a sociological dimension, you can understand any given codebase on a much deeper level.

A shared reality distortion field, that when powerful enough, can put an actual dent in the universe.

Powerful but unstable, potentially supercritical.

It's the emperor has no clothes kind of system.

Can be collapsed in an instant with one child laughing.

They'll look like non-dead ends, but are dead ends in dimensions you can't even see.

If you've met me in person you know that I have a habit of writing down notes live on my phone in a conversation.^[lg]

But if I don't capture them in the moment and context, they'll be harder for me to extract later, even if I had a transcript of them.

It's much harder to digest insights after the fact if they haven't already been a bit predigested already.

You don't have the touchpoints of "this thing you already are familiar with, but with this small tweak."

The minimal viable explanation has more steps in it.

Each step leads to a super-linear degradation of likelihood being received.