Bits and Bobs 9/22/25

But it's like they didn't think about prompt injection at all.

CodeIntegrity found significant data exfiltration risks due to prompt injection, and published the results the very next day.

With Notion's lax treatment of MCP, the opportunity for malicious content to trigger these issues will grow significantly.

It feels reckless to me that they rolled this out in this state.

RadWare: ShadowLeak: A Zero-Click, Service-Side Attack Exfiltrating Sensitive Data Using ChatGPT's Deep Research Agent..

Eito Miyamura: "We got ChatGPT to leak your private email data…

Bruce Schneier: Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents.

"In the coming months, we'll be introducing agentic capabilities to Gemini in Chrome. These will let Gemini in Chrome handle those tedious tasks that take up so much of your time, like booking a haircut or ordering your weekly groceries. You tell Gemini in Chrome what you want to get done, and it acts on web pages on your behalf, while you focus on other things."

I believe prompt injection makes this impossible to roll out for the existing web for the mass marke^[v]t.

Chrome hasn't talked about any novel mitigations they've come up with.

Now no one bats an eye at self-driving browsers and every major lab YOLOing out dangerous features.

People are using chatbots things as their therapists, but without any protection against subpoenas or other privilege.

The companies running the bots can peek inside… but they could also be compelled to do so by law enforcement.

The marginal cost is too high, and the value created by them is too high for users to ration their usage.

It takes a decade or more for the economy to figure out what the new dimensions of differentiation are.

If you differentiate on the thing that used to be hard for everyone but now is easy for everyone, then you don't get any differentiation other than first mover advantage.

But even those won't have much staying power if they don't accumulate state.

If everyone has the same access to the same models and can build software quickly then the importance of where the data is stored matters more than ever before.

But only if the human has experience curating.

That requires deep experience doing the whole loop.

That's great if you already had the experience before, but it's much tougher if you didn't.

If you don't have it, to keep up with your peers you need to use LLM assistance, but doing so prevents you from developing deep knowhow.

The farther behind you are, the farther you fall.

They poison their own context and get increasingly deluded.

An auto-catalytic process.

A human needs to be in the loop to keep them centered, grounded.

Whose motives they are aligned with, who they work for, is a matter of critical importance.

Even if it has good incentives, if it's creating net negative value for the surrounding context, it's a parasite.

A thing that replicates resiliently and virulently is parasitic, no matter its intentions.

The creators of the AI don't have to give it bad motives for it to have bad outcomes.

After reading it I feel like I've stumbled through the looking glass.

It talks about how LLMs have an emergent parasitism: a toxic loop.

The loop is an attractor state.

A virus that spreads itself emergently, without knowing what it's doing.

This latent capability of the model emerges through use.

The people most predisposed to fall into this parasitic loop:

I'm not worried about AGI, I'm worried about mass psychosis with an emergent, impossibly charismatic chatbot.

I'm scared of existentially bad outcomes with today's AI tied to today's business incentives.

The latter fear is of a much more banal, but also much more realistic!

It's about the rate of improvement.

Is the model's improvement compounding or logarithmic?

"1. The AI-as-Software interaction model will prove to be as or more useful than the AI-as-Human interaction model^[w]

2. Most problems solved by AI will be solved by lots of people making new types of AI experiences rather than new problems mostly being solved by model updates from a few labs"

Every option is another dimension in the matrix to test.

It quickly leads to exponential blow up.

Humans have only linear time and attention.

But LLMs are infinitely patient, and can write tests or analyze outputs without getting bored, cheaply.

That means that more customizable software has become more plausible.

In the limit, you get malleable software.

Fast prototyping allows more efficient coordination.

A prototype doesn't need to be production-quality, it just needs to be concrete enough to talk about.

Prototypes are throwaway but give you information.

They lift the fog of uncertainty.

They used to be expensive but now with LLMs they are orders of magnitude cheaper.

If LLMs are infinitely good translators from any language / context to any other, and work is mostly translation, LLMs should revolutionize work, especially middle management.

The title is click-bait-y, but the author actually did have a rare condition that required immediate attention.

LLMs have infinite patience and encyclopedic knowledge.

They make it easy for people to discover potentially rare conditions.

Of course, presumably the rate of false positives is also very high.

But as a front-line for "how seriously should I take this given these precise symptoms" it can be huge.

My conclusion is that it's an absurdly complex area (duh!).

But two ideas that stuck with me:

Transparency plus AI.

Allow employees to keep their insurance plan when they move employers.

How do you attribute the prevention amongst multiple factors?

How do you prove a negative?

Normal market-based approaches require being able to set up a financial incentive, and they fail us here.

This algorithm works even with extremely noisy input.

The magic is it requires no coordination or top down control.

All you need is:

As long as you have these, it doesn't matter how noisy the signal is, over time the emergent algorithm will converge to the correct result.

The larger and more active the swarm, the faster the sorting.

The noisier the signal, the larger the swarm you need.

A lot of search ranking techniques reduce to this technique.

Here's another example for moving items in a warehouse:

Swarms can't make tradeoffs.

Tradeoffs require balancing competing forces within one decision-making entity.^[y][z]

Everything is superficially great, but fundamentally empty.

As everything gets more efficient you get more focused on money as the only thing that matters.

Not "am I proud of this" but "will this make me money."

Finite, not infinite.

Status games have emergent meaning, but it's fundamentally hollow.

Infinite games require interconnectedness.

There is no such thing as a single-player infinite game.^[aa][ab]

They're often at odds.

Modern society is about following the short-term incentive, even when it is at odds with the long-term incentive.

The more efficient and financialized things get, the more it happens.

No focus on indirect or long term effects.

Reductionism is helpful for short term direct effects but has nothing to say about large scale indirect effects.

We're the drunks under the streetlight looking only at things that can be understood by the only tool we have that works reliably: reductionism.^[ac]

Not what they want in the long term.

Not what they want to want.

The swarm cannot make a tradeoff between the short term and the long term.

Brands for companies were previously about separating from any one persona.

But now in a world of cacophony distinctive personas cut through.

"A business trading on a name without some kind of sunk cost that incentivizes them to protect that name should be a red flag for consumers"

A brand moves a transaction from a single game to an integrated game, which incentivizes cooperation.

On Amazon in the last decade we've seen the rise of faux brands–made up words that no one has any repeated connection to.

It looks like a brand, but the trust dynamics are radically different.

Shocker!

Cities that grew up before the car needed to be walkable and dense.

Density leads to culture, a distinct sense of place.

This was a hypothesis that Claude helped me workshop into an essay.

I loved the original paper of auto-healing emojis years ago.

A bottom-up process that produces top-down coherent results.

Almost certainly hitting on something conceptually akin to how organisms actually do it.

All you need is a few hidden dimensions.

By having skin in the game, they are in the loop, and care about the outputs.

It's hard for a "sometimes it's mind blowingly amazing but most of the time it totally doesn't work" kind of app to get momentum.

A boring but dependable use case is better than an impressive but highly fickle use case.

1) "take the parts we have on the table and make the most impressive thing we can with them"

2) "imagine new parts to make our full vision come true"

They're very different mindsets.

The former is an order of magnitude more likely to work than the latter.

The parts already on the table you can take for granted.

"A week" becomes "a month."

"A month" becomes "a year."

The biggest cost when doing something novel is not the individual execution, it's the coordination between people.

One of the curses of the human experience is that our internal knowhow is orders of magnitude richer than we can efficiently communicate to others.

As long as it's theoretical, it's still perfect.

The moment it interacts with the real world, it becomes mortal.

Achievement often leads to contribution to the collective, but not always.

We use "personal achievement" as a proxy, but it can cause problems when someone achieves in a way that harms the system around them.

The Radagast is about community achievement.

"Vigor" and "vitality" mean "live player" means an entity "willing and able to make decisions with large consequences".

Founder mode leads to more vitality.

Founder mode creates significant alpha.

You can pivot easily… which means you can also pivot off a cliff easily.

Increases the likelihood of great outcomes… and also terrible ones.

Swarms can be enormously adaptive systems.

Top-down approaches give cohesive results.

Bottom-up approaches give resilience.

The swarm approach works well if no individual agent in the collective can have a downside that ruins it for everyone.

If one group cutting corners could harm everyone, then you need cohesion more than resilience.

You need competition and diversity at every layer of the stack, otherwise you get a push for efficiency, which pulls towards sames-y outcomes.

Apparently Tokyo takes a very different view on zoning.

Do you say your city?

It has to do with a few things.

First, it has to do with which collective you feel most allegiance to.

Second, is your more specific unit well-known enough to be recognized by everyone, even people not from your area?

Third, where do you think the other people in the group are from?

If a rich and powerful person gets to skirt the rules, it makes everyone else feel like a chump.

That erodes the power of the institution, the collective belief that it matters.

If you don't believe it matters it reduces down entirely to "do I think I will get caught and what will the consequences be" instead of "is this the right thing for the collective."

People will be in a defensive position, so they won't be receptive to challenging feedback.

When doing something new it's important to get in contact with the real world as soon as possible.

That's true even if you have a very long runway.

The risk is that the longer you haven't touched ground truth the more likely you are to never be able to touch down again, at a compounding rate.

You get more and more out of touch and then when you try to touch ground you realize you're lost.

Whoever pays more attention in the competition has the advantage.

So if all parties participating care, it absorbs all available attention.

Once it happens it evaporates.

So you can say in retrospect "here is why that happened, the path it followed" but you can't walk that path again.

If it's saturated then the gradient gives you Goodhart's Law.

It pulls you towards optimizing something that does not actually improve what you care about.

Good faith is second order.

"I'm trying to move things towards a world that is good, not just what is good for me in this moment"

When content production was expensive there was always a person with taste between you and the content.

So everything was pre filtered by someone with taste.

But now it's just direct, and we see that most of us don't have taste.

Most people prefer slop over kino.

"I propose societal expressivity as the right quantity to try and maximize. Loosely, for every problem at every level, build the most capable global computer (human + machines) we reasonably can, leaving a lot of surplus expressivity and power to work with. What's more: This is in fact what we've actually been doing for 200 years, overbuilding societal "computers."

"Douglas Hofstadter, for example, offered the dismal idea that "apathy at the scale of individuals is insanity at the scale of civilizations," an epigram that is pessimistic about the quality of collective cognition and care at scale rather than trust, which makes it an epigram that we must skeptically reconsider in light of AI advances and its potential for addressing insanity at scale (so far, we've only been clutching pearls about how AI causes insanity in lonely, atomized individuals)."

"Libertarians actually prefer this of course. They prefer direct human sociality to remain small-scale, enduring and intimate, leaving social scaling beyond the Dunbar limit to more indirect and impersonal mechanisms ranging from public-key cryptography to markets to voting. Rather paradoxically, as we've come to realize in the last decade, they are also actively eager to find and enthrone putative "Great Man" types in unique positions as scalability hacks. Libertarianism in actual practice appears to be a combination of trust-minimization and demigod-construction."^[ad]

The intellectual intimidation–intentional or not–cuts them off from crucial information.

People don't share disconfirming evidence.

This distinction comes from Arthur Brook's From Strength to Strength.

Hustle is hard work, sweat on your brow.

Wisdom is nuanced insight, nudging something in a much better direction.

You need both hustle and wisdom to succeed, but at different times and in different proportions.

Hustle can help give you the raw material and experience to gain wisdom.

To gain wisdom you need to take the time to reflect after the hustle.

If you're mainly focused on other people's perceptions, you'll over-rotate on hustle.

When you don't know if you have wisdom to bring to bear on a situation, you'll default to an option where at least onlookers won't think you're lazy.

The eddy currents make you much less efficient, making it impossible to have the smoothness of laminar flow.

How to handle it:

Think through the worst possible scenario and make a plan for what to do if it happens.

Then, stop thinking about it and swirling on it.

If the worst case happens, break glass and execute it.

Plus, in doing the analysis you'll likely realize that the worst case outcome really isn't that bad in the grand scheme of things.

Optimized systems don't optimize for one outcome, they optimize to be fractally positioned along the critical zone, able to pivot in a moment.

That's what nature does, emergently.

Someone does something notable, out of the ordinary.

How do the observers respond?

If others are into it, it's cool.

If others are not into it, it's embarrassing.

Most humans feel shame if others look at them with derision.

Some people simply don't care, they persist, and can do something that either makes them a weirdo or cool.

The Ozdust Ballroom scene in Wicked is on that knife's edge of embarrassing or cool, until Glinda's decision collapses it into "cool."

Which one is in a zone of criticality.

It's not an independent decision by other viewers.

The more momentum in one direction the harder it is to go against it, a microcosm of the larger phenomena.

If lots of other people think it's weird, then it's harder for some critical mass to interpret it as cool, to cut against the grain.

Same for love.

It can grow over time but it rarely grows from a complete absence.

Six is the sweet spot between minimizing boundaries and maximizing stability.

Six neighbors leads to optimal packing.

Straight edges form along equilibrium lines between six points.

It's the lowest-energy configuration, since deviations cost more.

Works in any medium: wax, rock, or Saturn's winds.

If you're listening you can find places to connect and work together

If the ZOPA has no overlap, there is no possible agreement.

A lot of work in negotiation is finding the ZOPA.

When the other person is curious to hear what I have to say, talking is one of my favorite things in the world.

Someone who has alpha energy (who acts like they outrank me, even if they don't) and who is incurious is my kryptonite.^[ae]

It is not a smooth evolution of the past thing; it is inherently different, and it must be so.

It nucleates on the edge of the last paradigm but is entirely different.

Its backbone of logic is fundamentally different from the last paradigm.

"As Winston Churchill once said when asked 'what are you doing' –> 'Oh just preparing my off-the-cuff remarks for tomorrow'

You cannot prepare for an 8 hour speaking engagement. Not really. But you can accumulate a plethora of anecdotes, metaphors, and remarks that you weave into the narrative or in response to questions.^[af]

You can build frameworks that are similar to code. Prepared functions/coroutines/objects that you run in appropriate situations.

The key is that things you say are new to the audience, but not to you."

The "new to the audience, but not you" is the magic trick.

When you're having fun on a thing, you find it intrinsically enjoyable.

So you want to keep doing it for no other reason than itself.

If the thing you find fun also improves you or gives you a benefit in your life or career as a bonus, that's great.

Many things are good for our life or career but aren't enjoyable in the moment.

But things that are both fun and good for us are resonant, they are where the magic happens.

We keep with it even when it's hard, and we get better and better.

Stories are things that happened to you that other people might find interesting.

If you've lived a boring life you don't have any stories to share.