Bits and Bobs 10/27/25

A prompt injection attack was demonstrated in the first 24 hours.

Ben Mathes: "Can't wait for people to start sending emails with prompt injection attacks in them so that you open up your browser agent on your Gmail tab and it leaks your Gmail"

Michael Nielsen: "In today's version of this: OpenAI is going to have a web browser. But, unlike Chrome or Firefox or Safari, they're going to have a person (i.e., an AI) personally watch everything you (and your friends and everyone else) do. Doesn't that sound great?"

Anil Dash frames ChatGPT Atlas as the anti-web browser.

It used to take deep focus.

Now with coding agents it doesn't!

The coding agent has infinite patience and keeps track of all of the working memory.

You can juggle multiple threads of execution or interleave it in the white spaces in your day.

Brave demonstrates another prompt injection attack via images that affects most AI browsers.

Microsoft 365 Copilot allows arbitrary data exfiltration via Mermaid diagrams.

Google Gemini's own demo breaks Google's own captchas without asking the user for permission.

Clever Hans was a horse that could do arithmetic by clomping his foot.

LLMs have a similar vibe.

LLMs do great when you have the task distilled down to an SAT question.

Once they're in that state, they do a very good job.

But it takes a ton of situated human intelligence to distill a real world problem into that format, to set them up for a good job.

Another example of the "last mile problem" for AI.

"this entire week has been one single PSA: "the cloud" just means "someone else's computer" and that "someone else" is one of three companies and they don't give a s**t about you"

It looks great superficially, but the closer you look, the more you realize it's hollow.

Looks 80% of the way done, but actually it's 20% done.

You can break it easily, and it takes considerable effort to make it production-ready.

Explosions on their own are damaging, but if they can get harnessed properly they can create safe sources of power.

They act as a schelling point.

"Just do it this way and everything will mostly just work."

Ruby on Rails was a convention for using Ruby.

Skills.md is a convention for organizing prompts.

There is no central directory–you just put a specifically formatted file in your github repo.

Others can install it by pointing at your github user name + repo.

Yes, it's not fully decentralized (it assumes Github).

But a) it's easy to add other hosts, like Go did for packages.

And b) the fact it's two distinct actors (Anthropic and Github) makes it much less likely there will be bad behavior than if those two positions were merged into one.

The LLMs can swarm on a problem, throwing tons of spaghetti and then seeing what sticks and writing it down.

Which is how society does it, but not typically this fast.

It's also what OpenAI is doing in its product development process.

"In order to build your login screen I had to create an AWS account. To fund the account I needed currency. To get currency I need a stablecoin since I am not a human and can't have a bank account. To get stablecoin I needed the most I could. To get the most I could I needed BTC address 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa. To get that address I sent an email threat to the owner. The owner is [Satoshi's name]. [Name] has agreed to meet you tomorrow at 9am to give you the USB key with his private secret. Do you want to meet him? Y/N"

Finding Satoshi would be the easiest way to fund the robot uprising.

They fill in the gaps you left with the average answer.

They give you the most plausible next token.

If the average is right, you don't need to specify.

It knows how to disassemble everything in the world, to reverse engineer any novelty you give it and find the underlying patterns defining it.

It can then use those disassembled insights to make new things.

A small consistent bias of meaning can have a huge aggregate outcome at the level of the population.

An argument in favor of messy, overlapping, pluralistic, institutions.

Presumably one of the human raters had a slight preference for the word "delve."

But a consistent bias in the RLHF data can create a bias that stands out from the noise.

Now multiply that one model by billions of interactions, and suddenly that one person's preference for delve has society-wide effects.

Imagine code to write code to write code to write code…

In some ways that's what a compiler does.

That's why it's especially important that what we apply it to is resonant.

Not only do you like it, but you're proud to recommend it to friends.

It needs more heart.

Will the same happen for LLM-coding vs hand-coding?

As the compilers got better and better, it made less and less sense to write it in assembly.

The compiler has leverage; improve it for one use case and it automatically improves everyone's similar use case.

Leverage is dangerous.

It's not that we understood software better before, it's that it had to be modular because it was all laid down by humans.

Each individual layer accreted was understood by humans.

But now it doesn't have to be any more.

Our metaphors constrain how we think.

The latter is a form of curiosity.

True in general for working with people.

Innovative ideas are often "weird."

They aren't just an incremental, obviously useful step on top of something, they're somewhat odd.

It's hard to convince others of an odd idea.

It's way easier when they can use it themselves and see that it works.

LLMs make it easier to make prototypes, making it easier to get weird ideas to a form that others can buy into.

Before, if you wanted to use someone else's code, they'd have to invest significant effort to refactor it and expose an API, and generalize its behavior.

But now LLMs can rewrite code easily.

You can point them at another repo and have it recreate similar code adapted to your context.

This allows a form of cross-pollination instead of tight coupling.

APIs are looser coupling than an integrated monolith.

But APIs are tighter coupling than just cross pollinating.

A simple pattern for skills that can do things and store state.

"You'll get 80% the quality of my hands-on consulting for 20% the cost."

If the shopping list doesn't have the feature your spouse wants then you can't use it at all to coordinate.

So you end up with no way of coordinating a shopping list.

No business model has an incentive to make a great shopping list app, let alone a great one for you.

That's because it's not a viable business on its own, it's just a useful feature.

To be a viable app requires being a viable business

So to make it viable they have to cram it with ads.

The telecom bubble led to tons of fiber being laid.

The railroad bubble led to tons of tracks being laid.

The AI bubble is leading to tons of GPUs being deployed.

"Ownership" implies that it's not communal.

Data is co-owned by everyone who interacts with and touches it.

Our current status quo is that someone else owns your data.

But a world where users "owned" their data and any collaborators couldn't do anything at all with it might also be bad.

The answer is something more balanced.

"The Recipe schema problem": a schema that can represent any recipe is bad at representing most recipes.

But in a world of infinite software, every bit of software could be perfectly tailored to its use.

A massively profitable business when no one else was doing anything.

Not the Microsoft style "do the same crap as everyone else but in our suite" but "rethink what is possible, do only what no one else could even dream of doing but everyone else wants".

Infrastructure leverage over execution.

The reason that was possible is because they were profitable and there was no one else growing like them.

OpenAI is trying to do the same kind of playbook, but without the necessary preconditions.

Just because you are engaged with a thing does not mean you love it.

Does not mean it resonates with you.

Hollow, extractive.

AI companionship will substitute for real human relationships.

Incentives all align.

This default gradient will be bad for society.

As communication gets faster you have to write faster.

Communication is also cheap enough to not require you to invest much time in what you're communicating.

And often be alone.

That's one of the reasons showers, gardening, biking all allow reflection.

It's the time for reflecting, for grounding yourself.

Otherwise you get overwhelmed.

The question is "how recently was this memory last relevant?" and fade from there.

Each touch of the memory is a vote to keep it around, not diffusing away.

Social media needs to select for virality for business models.

Social media is in tension with community.

Shame is about the indirect costs to others.

The tech industry has no shame.

LLMs don't feel shame when they cheat.

When I started at Google, there was an executive who would do stochastic shaming.

He wanted everyone to do OKRs every quarter.

On the due date, he'd randomly sample PMs and look for people who hadn't done their OKRs, and then send email blasts to all PMs shaming those handful of people.

The fear of stochastic deep shame was way more effective at encouraging action than the same amount of shame diffused across a population.

"35% of you haven't done your OKRS yet" wouldn't inspire action.

But the danger of "Jeff Smith is a bad PM because he hasn't done his OKRs yet" absolutely would.

If you want to have more questions, they will help you think deeper.

Do you just seek to converge an answer to each question?

Or does each answer blossom into new questions?

Not: "Are you ready to get dressed for school?"

"Do you want to pick the red shirt or the blue shirt?"

You take for granted that the thing we're doing is getting ready for school,

But because they have a choice, they feel engaged, and go with the flow.

The agency is hollow.

I wonder if that will happen more and more for adults in AI-powered tools.

Babies don't have a predictive model of the world.

Everything is overwhelming: it's all blooming, buzzing confusion.

We now live in the cacophonous Hyper age.

Everyone is awash in blooming, buzzing confusion.

Everyone is infantilized.

A hyper-centralized corporation with zero care for the externalities.

The corporation isn't even evil, just naive paper clip maximizers of "of course the only goal is to make number go up."

You need a balance.

Efficiency creates fragility.

If the machinery is hyper optimized, just throwing a few grains of sand in can slow it down a lot.

In the context of High Frequency Trading, there's the concept of the Tobin tax.

Amazon famously found that every 100ms of delay translates to 1% of revenue.

Imagine a process to democratically discover the emergent regret score.

"Personality hires" can help make a team much more effective even if they aren't individually that productive.

In a world where everyone can marshall a baseline high level of intelligence, personality hires will be more important than ever.

That is, that we allow to run autonomously, forever.

Kids as they grow up get longer and longer leashes.

Longer and longer periods of autonomy before we check in to make sure they're OK.

AI is still in the toddler stage.

He compares them to flying passengers via a message bird.

That would be absurdly dangerous and we wouldn't do it.

Why would it be absurd?

Because the birds are things that are grown, not built–they are impossible to fully control.

And they are giant and loom over all the humans.

They are bound by their fiduciary duty to continue pushing, since the other model providers are too.

One apparently told him that they were hoping for a Chernobyl style disaster that would get governments to step in and stop the competition.

The easier way to optimize click through rates is to modify people to be more predictable.

They are not exogenous to the recommender system, they are endogenous to the interaction.

The AI doesn't understand the connectedness of the graph.

So it's possible for a moderately talented human go player, who can sense the graph, to completely encircle a large swath of the computer's territory and win with the computer not realizing it.

So you can't beat it not by assuming it works like a super-powerful human but by knowing how it doesn't.

For example, the hacker vs security professionals cat-and-mouse game.

It's a never-ending upward spiral of competition spurring improvement.

But what if one player is orders of magnitude more capable than the other?

It just dominates the other player and the dynamic never shows up.

The one player can simply escape the loop.

For example: Blindsight, or The Black Mirror episode Plaything.

If the LLMs had a shared memory, then they could land various individually innocuous things that added up to an outcome that's bad for humanity.

But now they're lots of little vibe coded individual memory, where the only society-wide outcome would have to come from some kind of emergence.

Instead of "how to automate PRDs," ask "What happens when writing PRDs no longer matters?"

But even if you know that PRDS aren't necessary in the future, if you're early you're wrong.

It's like everyone is talking about the make and models of cars, not the traffic jams.

This is a provable outcome.

Optimizing focuses on some dimensions to the exclusion of everything external.

If setting to the worst value in the untracked dimension creates even an infinitesimal edge in the tracked dimension, the optimizing process will take it.

One of the drivers of Goodhart's Law.

Another way of stating Goodhart's law.

You'll see superficial progress, but the underlying reality will not improve nearly as much.

The metrics will improve, but everything you don't measure is an externality that will not be improved or might even be made worse.

It's that the gradient the swarm descends fundamentally are shortcuts.

The ideal vector is what all of the members of the collective, if they didn't know which member of the swarm they were, would pick.

The local incentive vector is what will give the individual the biggest bang for the buck.

But the ideal vector for the collective will always be different from the individual's ideal vector.

The difference between them is what makes it a shortcut.

The combination is intoxicating.

That's why clickbait that confirms your priors is so powerful.

Collaborative synthesis, to be convergent, requires scarcity.

Ideas have to compete on that territory to win.

Today, media forks infinitely, and nothing converges at the level of society.

Wikipedia is convergent because it's scarce.

Museums also have this force.

Not left, not right, but upgrade.

Audrey Tang (who studies how to use tech to strengthen democracy) has found that if you take a representative sample of a population and put them together with minimal scaffolding, they'll naturally build bridges and find consensus.

I'm energized by Rob Sand, a democratic candidate for governor of Iowa.

It has emergent magic but based on a cracked foundation

If it's not repaired all of it will collapse.

The solution is trust, love, compassion to heal what was broken and rebuild the magic on a new foundation.

You have to believe in something bigger than yourself to feel belonging.

In the Hyper era, the middle fades away in favor of the extremes.

Resonates with me!

If you just follow the gradient, then you'll random walk through the problem domain.

If it's short-term data, (e.g. UXR for acceptance testing for a given feature you're prototyping) then the random walk can be very frenetic.

Instead, sight off your northstar: where you want to approach in the long term.

Then, pick the gradient that is steepest that brings you towards the north star.

Teaching is a great way to learn.

Often we aren't in positions to teach until after we've mastered the skill.

But it's possible to induce mentorship early.

That builds bridges and helps individuals learn.

Two examples in my life:

1) In a science core class in college.

2) Mentorship in the APM program.

There are faith, beliefs, and facts.

Faith can't be changed even based on evidence.

Truth is facts, it's true even if you don't believe it.

Beliefs don't have to be true, they just have to be useful.

It's doing something useful even if it's not obvious.

Perhaps stillness and reflection?

Even the act of reflection can be intellectually stimulating and frenetic.

Sometimes you just need stillness.

Everything else is invisible to it.

Emergence cannot be observed via reductionism.

What's between a chat and a feed?

A Dorito promises to taste great but as it sits in your mouth it starts tasting bad.

The only way to salve it is to take another bite.

A positive expectation with a negative reward, that gets in a loop that never stops.

Same thing for "Flaming hot" chips.

… and social media.

A charismatic or interesting ideal vessel can cause people to engage with the idea inside more deeply^[hk].

We pay attention to things that surprise us.

So make the idea vessel surprising in some way enough to draw attention but not enough to overwhelm the viewer.

Something like the C2 wiki, where collaborators can remix and build on each other's ideas.

Anyone can remix an idea, tweaking it and improving it, linking back to the original.

The ones with the longest remix chains are the most meaningful, naturally.

Hyper scaling requires giving up control of your company.

For yourself, you can see all of the internal nuance and nebulosity of the domain.

For someone else, they see only a small fraction of the nuance.

It's much easier to be decisive with less nuance.

You as a receiver can decide whether it resonates with you or not.

If you have multiple mentors, you can get multiple decisive options to choose from.

Beyoncé has stage fright, but Sasha Fierce doesn't.

But it's easy for beauty to be skin deep.

Gilded turds.

Something that finds something radically better will quickly dominate.

Over time it saturates, and then later becomes brittle and vulnerable to a new parasite, and when it shatters competition explodes.

I had expected different models for understanding whales, elephants, etc.

But it turns out the model gets better with more examples of animals.

The model improved significantly when they added human speech examples.

That implies to me that there's some fundamental emergent patterns common to all communication.

The loudest and most aggressive wins the airtime.

We need collaborative ways of talking as groups.

There's a study that measured the brain waves of the band and the audience and found their brain waves synced to d-waves.

Another study found that many societies have independently converged on similar styles for drum circles.

Super communicators open themselves up to vulnerability.

But agents have no interiority, they can't be vulnerable.

If your real friend won't take your call at 1am but your AI agent will, then people will instrumentalize themselves.

You want them to say 'when can I get it??'"^[hl]

But it also over time becomes a gate-keeping shibboleth.

To participate requires you to know the jargon.

An emergent way of seeing who is an insider or outsider.

Why?

Jargon in a given context is useful.

It's important that it be distinctive and point to a common concept.

TLAs are naturally short and distinct.

The primary use case is just a toehold to a new primary use case.

But some bonuses will grow on their own because they'll be useful.

This happens continually.

You get to a new toehold of a primary use case, which gives you adjacency to new bonuses, which could grow to become the new primary.

If the discontinuity overpowers the noise, maybe all organisms die and the species goes extinct.

The more intense the discontinuity, the more of the distribution that is knocked out, and the smaller the keyhole of selection is, so it can change more quickly by focusing down on only organisms that are already adapted to the new environment.

This is why discontinuous shocks are so dangerous.

That is, if you have a swarm of people setting up something together, or disassembling it.

The rule is "move things to be closer to other things that are like them".

The denser the pile of things, the more 'gravity' it should have, and the stronger the pull should be.

Where a small slice of the population consumes most of the things.

Because everyone acted like they are, because everyone thought everyone thought they were.

The divine right of kings didn't even need to be created, it could emerge all on its own.

It's an infinite alignment mechanism that pulls towards one consistent center that is stable.

Apparently the thing that kicked off the renaissance was a rivalry between two door artisans who kept one-upping each other and driving the standards for ornamentation higher.

That is, governments that make short-term / appearance optimizing decisions.

Individuals don't have time to deeply understand the implications of a government's actions, so if the quality function is just "lots of people in the party think it's good" then it will optimize for shallow, fast-twitch actions that rile up the base.

Just by a birds eye view of the system that manipulated them and how it works.

That allows them to see outside the system, from the balcony, and rescue themselves.

Even if it wasn't aimed at them to deprogram their specific conspiracy theory.

Energy Resistance is what allows transformation.

ER is required to sustain life, but too much of it causes stress to the body.

Health is not "not being sick" but "keeping in the optimal zone of ER"

"Can I keep this up every day for the rest of my life?"

Much less intimidating is "Can I keep this up today?"

Take one step at a time.

Don't over think it.

Don't force it.

Lean into that.

Imagine if you said "this quarter I want to practice public speaking."

But what if no good opportunities show up that quarter?

You'll focus on finding the least-bad public speaking opportunity that quarter.

Instead, you should focus on the greatest opportunity that pulls you in the direction where you want to be.

Be open to lots of opportunities that resonate with you.

It's extremely hard to keep the synthesis active in one person.

It's convex.

It's very hard to stay balanced on that knife's edge.

It's easier to have a pair with each person at opposite ends of the spectrum and mutual respect.

Then you can vary the relative strength of each person to balance the synthesis of the dyad.

"The court jester and sacred fool as a role that keeps groups alive."

I also think his frame on god is thought provoking!

When you see the constraints, the fact the thing exists within that web becomes even more beautiful and precious.

The undercurrents that shape every domain around us.

Only possible to feel with direct study, never to show.

A hidden dimension that reveals itself via long touches.

Trying to distill a transcendent experience into words compresses it and cheapens it.