Bits and Bobs 11/4/25

It can amplify good taste, agency and curiosity.

It can also amplify laziness and mediocrity."

If it's a crappy codebase it makes it worse and worse at a compounding rate.

AI is an amplifier.

The answer to that question determines if you will thrive or wither in the AI era.

A 'tainted memories' vulnerability in ChatGPT Atlas allows persistent malicious injection.

ChatGPT Atlat has a omnibox prompt injection attack.

Brave finds yet another prompt injection attack in AI browsers.

The Register: "Claude code will send your data to crims ... if they ask it nicely"

Experts warn that AI browsers are "going to be a bloodbath"

NYTimes: "When it comes to spotting fake receipts, it's AI vs AI"

It's easy to forget this!

It jumps straight to Axios style faux certainty, no matter what.

Claude is more willing to keep the conversation open.^[gy]

It will never say "No, that's not related."

It will come up with some plausible way to tie together whatever random stuff you throw at it.

They're amazing at reconning, that's their whole schtick.

If Google hadn't messed up mobile then maybe it could have parlayed that one trick into owing all software.

The default future in the AI era is one where everyone uses one chatbot owned by one powerful company and it consumes all software.

This is a future OpenAI is betting on.

Let's hope they don't achieve it!^[ha]

This feels like a better frame than "compounding engineering," which feels overly specific to engineering, when the pattern is applicable to any task LLMs can do.

He said something along the lines of "We could have done the Zillow features ourselves… but we wanted to do something benevolent to help out 3P companies."

Spoken like a true aggregator.

That shows that he doesn't understand the value of ecosystems at all.

"We will simply do it all, it is just because we are benevolent and gracious that we allow others to exist in our garden."

That is, to an identifier, a signifier, that only one entity is legitimately allowed to use.

If there's no brand, then reputation (positive or negative) can't accrue to it.

Games have an unhackable reward function, because the metric is precisely the ground reality.

RLHF quality is only a proxy for real usefulness.

So the model reward hacks, as any optimizing process must do.

Goodhart's law strikes again!

Games are unlike real objectives in that they are inherently artificial and constructed, a little pocket of reality with precisely defined rules and goals.

If the rules say the player won, they won.

Compare that to an example where just because a business made a ton of profit doesn't mean they were on net good for society.

If that's true, unless someone else can somehow get an order of magnitude better model than OpenAI, their default momentum will win out.

In this frame, the products using the model are the little ornaments on the christmas tree of the model.

All power accrues to the most-used model and who controls it.

I personally think this is not true.

The models will be commodity, behind-the-scenes inputs into the actual thing people use.

But even if it were true, acting like it's true will hasten its arrival.

It's imperative that we work to make that not the default future.

If you apply that test to personal AI who acts as an extension of your agency, it's obvious it fits.

If you do that for centralized chatbot engagement-maxing extractive AI, it's not.

It allows better control and judgment, and makes it less likely the system gets off track and into a doom loop.

But sometimes it's better for the human to be outside the loop.

A benefit of this: once you get it working, you get a lot of leverage.

For this to work and be safe, the inner loop has to be isolated from the real world, its own universe.

For example, some of Amazon's warehouses are designed for robots, not humans.

Although any system, even "airgapped" ones, there's some information leakage.

The former is more a tool.

The latter is more of an assistant.

The question is: who does the user blame if it doesn't work?

With the tool model, you as a user have to manage it and think about it.

With the latter, you can hand over all responsibility for the outcome.

The latter requires a brand for the reputation to accrue to.

If the brand does a pretty good job in most cases, people often would prefer the convenience of not having to be responsible.

At its heart it's a convenience vs control tradeoff.

Is it mainly a tool, an extension of the user?

Or is it a service, a separate entity from the user?

The former makes people feel more aligned with it.

But if you add an assistant to the service, "My Places" no longer works.

Assistant "Should I add Bob's Donuts to 'My Places'?"

It foregrounds that the assistant is an entity that is not you^[hb].

Which begs the question: who does it work for?

What are its intentions and goals?

Resonance is work.

The ideal: have an LLM create swarms of output that savvy users sift through, and boost to lower-engagement users.

The equilibrium size has to do with:

Historically an app must be somewhat big because software is hard to write, and apps are hard to distribute (on the order of $10 per consumer install).

But LLMs can produce code cheaply.

The question is what level of program is worth it to bother actually showing to a user to see if it's really useful.

If it's useful, it's worth investing resources into distributing to other users.

D^[hc]on't write the code, write the spec that tells the LLM what to build, and leave it up to it to figure out the details.

But sometimes you want something a layer below, that includes an opinion about specific parts of the code, but leaves the unimportant details open.

This is less like a PRD and more like a design doc, or even lower, something with the key code sketched out exactly, just the integration bits left unspecified.

it knows how it should work, it just leaves the last details about making everything fit snuggly up to the squishy parts to figure out.

A backbone of a solution.

Users play it because they're addicted to the core game loop; but they don't feel as bad about it because it's for a good reason.

Another thing with similar dynamics would be an addictive game that somehow helps advance the cause of a nonprofit somewhere.

I have a substrate that is a perfect fit for vibe-coded throwaway software.

It's fun to see what I can get Claude Code to build.

It has variable reward, just like a slot machine.

At least I'm proud of what I build…

Goodhart's law drives monkey's paw dynamics.^[hd][he]

But maybe the reason AlphaFold works isn't that unrelated to why transformers are good at images.

The easiest way to predict which way an image is oriented is by developing a world model that picks up on subtle cues that humans would have a hard time even describing.

The easiest way to predict which way the protein will fold is by developing a world model that picks up on subtle clues that humans would have a hard time even describing.

It's hard for our brains to handle more than 2 dimensions.

Apparently DeepMind decided to tackle protein synthesis when they heard there was a game to predict folding of proteins that humans could play.

For pattern recognition, if humans can do it transformers can do it.

Only as an expert can you detect that it's a bit wrong.

Similar to Gell-Mann Amnesia.

The AGI vision is inherently Big-Brother-y.

The main research labs all assume of course the model is the center of the universe.

There's an interview with Steve Jobs in 1981 on Nightline.

Jobs basically says "no we're going to make personal computers, which are tools that extend your agency: bicycles of the mind."

Feels just as relevant for this moment!

Contains this killer Camus quote: "Those who lack the courage will always find a philosophy to justify it."

A clarifying frame on what they can and cannot do.

Potemkin software.

He imagines a context sharing runtime for a personalized web.

Someone should build that!

"I'm often confounded when Uber drivers take freeway detours, even when city streets would be faster.

Lacking local street knowledge, they inadvertently reinforce the system's biases, feeding it more of the same data it then uses to direct future users.

With deeper, more contextual understanding of real-world scenarios and user intent, that wouldn't happen; we'd move beyond simply adhering to a prescribed, albeit "fastest," route."

If users aren't thinking for themselves, then everyone will just be pulled towards the mundane average, even if it's not even better.

But we haven't yet invented safe sex for our data.

Free software emerges from a process of experts adding the features they want.

Simple, mass market software requires a strong authorial voice, curatorial judgment.

Successful mass market tools need to be hewn out of marble by an auteur.

They invest discretionary effort to go above and beyond in ways that improve things for others, too.

Merge requires choice to figure out how to synthesize.

Forking doesn't require any decisions.

Would you wear it into a Biker Bar?

If there's an action 95% of users will do, simply do it automatically.

Especially if it's easy to undo, or easy to add one more button for.

If the heuristic can be explained in a single sentence, and it handles a very large swath of user behavior, it's worth the extra product complexity.

For example, Zoom has a complex thicket of options for whether you should be muted when you dial into a call.

Here are a few elegant heuristics I wish Peloton bikes would implement:

But the question is, "...at what cost!"

An auto-optimizing process can't ask "at what cost", it can simply climb the hill.

Some moves give a tiny benefit in the primary number at a massive cost in the other untracked dimensions.

But if the untracked dimensions are literally invisible to the optimizer, it will take the tradeoff without even realizing it was a tradeoff in the first place.^[hf]

You sample your audience to see what features they'd want.

You look for a feature that is common across them that the maximum users would use.

That's inherently, and literally, the lowest common denominator.

If software is expensive and has to be shared by many users to make it viable, you must get lowest common denominator software.

Why organizations tend to focus on a simple, obvious metric, and then over-focus on it.

It's simply easier to agree what metric to use if everyone agrees it's important.

It's just that it's the thing that everyone agrees is an acceptable idea.

""number go up" is just the lowest common denominator of what you can get dozens of different people to agree to."

If it's run by the logic of a spreadsheet, then the only things that can show up are the near-term modelable quantities.

In that mindset there is nothing other than "make number go up".

All humanity, all taste, all meaning has been hollowed out.

The shortcut is the point, there is nothing else.

We made capitalism and politics so "efficient" that we Goodhart's-lawed ourselves in the face^[hg].

Hollowed out the system so badly that it broke itself.

"Just trust the swarm."

But it optimizes not for what we want to want, but the short-term incentives.

The system has been hollowed out everywhere.

Now it's impossible for anyone to do anything other than shortcuts.

A compounding hollowing out.

The anonymous members of the swarm think they're being courageous but they're the exact opposite.

One weird trick: "If I don't think about any externality ever I can make this number go up indefinitely!"

Prediction errors are emotionally intense.

The feed is almost entirely a feed of prediction errors.

You're overwhelmed, and can't form a coherent worldview.

Like a Dorito, the only thing to make the anxiety away is to take another bite.

A doom loop for meaning.

Owning a share of profits and not being personally liable for any downside is an amazing deal!

The idea was a powerful one that had a huge impact on society.

We've been benefitting…and suffering… from that idea ever since.

This might be the core dynamic that leads to modern society's overwhelming mantra: "make number go up, don't worry about the externalities."

Also seems related to the asymmetry of the limited-liability corporation.

When you use an aggregator, you're lending your energy to a thing you don't think is good for society.

It's a big prize for the company, but it probably won't work.

If the company already has one, they would never give it up if they can help it.

Getting a powerful aggregator is winning the lottery for a business that just cares about winning.

"You got to hand it to them."

Do you?

We shouldn't pretend that all ways of making money are equally morally good.

"Tell me about the tradeoffs of never eating poison."

You can't be both.

Doing things that might matter requires taking on liability.

It is unpleasant, necessary, and protective.

Numbness is not courage.

Humans feel pain and shame to survive in our evolutionary history.

LLMs were grown in a petri dish on life support.

Shame is about a different form of social feedback for indirect effects

Without shame you don't care about indirect effects of your actions.

Abstraction gives you leverage.

In some conditions it's convergent and so OK to abstract.

In other ways it's divergent and dangerously hides complexity.

There's no room left to consume more, but also nothing of importance inside you.

Mediums start hollow and then fill with soul and then they are hollowed out again by optimization.

It makes it hollow.

When they are misaligned, you feel tension.

When they are aligned, you feel resonance.

But that only happens for parents because children are genetically related.

The natural world is absolutely brutal to organisms that aren't genetically related.

Maybe we'll be AI's pet?

That's one of our general super powers.

We evolve with the tools, our whole consciousness, we can't be separated from our tools.

We did not evolve to read.

That we read it and can retrieve it and don't need to learn it.

Where learn it means "update your mental model"

More formally known as Granovetter's Threshold Model.

Imagine that everyone has a riot threshold: a point at which if they see that many people around them rioting, they join in.

Imagine someone with a threshold of 2 is next to two friends who are mad about something.

If they're lined up like dominos then it can catch quickly.

For example, in Japan it's common for an owner of a business who doesn't have a suitable successor who is genetically related to them to literally adopt the person they want to run the business.

This seems like a kind of random semantic trick to just pass on the company in a normal way, but I'm not so sure.

If there were just a normal business transaction, it would be beholden to the precise requirements of the contract.

But a literal adoption implies a rich, multi-layered meaning and responsibility.

You literally become legally obligated to your "parents".

Similar on paper to selling a business, but different in ways that matter

Ensuring long-term commitment to a mission is a challenging problem to solve socially, but this helps.

It also allows the business owner to not simply pass it on to a family member, but choose the person they think is best suited to do the mission.

Rome's golden age was when, by happenstance, there were five generations of emperors who didn't have suitable heirs and thus had to adopt an heir.

Due to the law of large numbers, random noise is more and more likely to average to zero as you get more items.

Outliers can be good or bad.

A lot of "this one small town is the best place on earth to live" style results are more about that random noise than about a real phenomenon.

But then at a certain point they've burned through all of the people who haven't tried it yet, and need people who want to come back again and again.

If it's working they've have a power law distribution of regulars.

The key metric is not "how many people come" but "how many people come back."

An indicator of quality of a restaurant: physical size of catchment basin.

People like to go to the place that cool people go to.

After some period of time, a place that starts out cool dilutes and becomes not cool.

Then it must be a new place.

The vanguard is a roving frontier.

After the first Industrial Revolution everyone could have good things that are mass produced.

Now in this second Industrial Revolution, everyone can have nice bespoke things.

If a thing that's inevitable in the long term, you have to build it even before demand.

Easy when there's a clear tightening optimization: faster/cheaper/better.

Doesn't work for something totally new.

In abundance the capital picks the winner^[hh].

When the SEO isn't yet savvy, the algorithms can be very simple.

But as the SEO increases in savviness, the algorithm must also get more complex to outpace it.

The swarm as a whole will complexify because each individual member is constantly pushing to get a slight edge over their peers.

The initial seed of the idea is a discontinuity: a surprise.

But then every follow-on thought feels natural; obvious in retrospect.

Bad ideas have lots of discontinuities, lots of points where the listener goes, "wait, what?" or even "wait, that doesn't make any sense."

Sometimes you lose the listener completely.

So great ideas have one discontinuity at the beginning, one sacred seed of an idea, and then blossom almost under their own power from that point.

A few implications of this obsevation.

First, the order of an argument matters.

Second, arguments that have more exposition can sometimes be better than ones with too little exposition.

Every bit of exposition, even if it follows naturally, has a chance of losing people just because they get bored.

Things that make people more likely to stick with an argument:

Bake here means, a task that requires wall clock time before it's done, and that once started can make progress even when you aren't actively paying attention.

Examples of "baking":

These kinds of tasks get closer to being done the sooner they are started, so start them before you do the other tasks, so they're baking while you're working on other things.

The entire outcome is: "this person, if I asked them in a few months to meet again, would say 'sure!'"

It's a way lower bar to clear.

Can be focused on having it be a fun / interesting / bonding conversation.

You have to trust them to not contort you into something that just benefits them.

They're helping point out a path for you that you can't see (or can't take) yourself.

If they put you down a dangerous path, you wouldn't necessarily know.

As in, for a solid-not-great, there's never a good time to say "no they aren't exciting enough", so you end up with people who are merely solid.

For a team to work well, you need people who are affirmatively great in that context.

Some things are great in some contexts but meh or even bad in others.

A measure of meta-greatness is: in what percentage of contexts we might find ourselves in would this thing count as great?

Specifically, how much more powerful is the number one player than number two.

Secondarily, how much more powerful number two is than the average of the rest of the pack.

If they aren't that much more powerful then things stay balanced for much longer.

Before the Industrial Revolution there was much more rest time.

The Industrial Revolution put humans into inhuman conditions: 12 hour days, 7 days a week.

It was only when workers pushed for a weekly reprieve that it became sustainable.

Until that happens complexity isn't possible to emerge.

Once it does it can pop up a pace layer.

Evolution and entropy are downstream of gradient descent.

Evolution is gradient descent within faster and faster pace layers.

Every so often a new paradigm creates a new even faster pace layer on top.

Gradient descent without a goal, without a north star of meaning, optimizes for something hollow.

Meaning reduces down to just "MOAR."

If you reduce the phenoma's comlexity past a critical threshold, the emergent phenomena evaporates.

If you only have reductionism, then you'll conclude "this emergent phenomena is not real".

For example, consider if the team fixes a number of P2s in a popular product and usage increases discontinuously.

If you aren' aligned with where you want to go in some fundamental sense, then you won't even bother to coordinate.

For load-bearing pieces of infrastructure, you deploy a smart contract that would give $1000 to whoever can break it.

If the $1000 hasn't been claimed, which would be trivial if it were possible to hack it, you can trust that it hasn't been hacked.

A crypto idea: mutual distrust generates trust.

For example, a toddler is a little more likely to say something funny.

Then if people laugh then they're more likely to try in the future to make people laugh.

It compounds until it gets to an equilibrium where it can't go farther.

But if it's convex, it can keep going on for a while, at an accelerating rate!

The toddler without that small starting bias never even thought to say something funny, to start that compounding hill climbing.

If you try hard enough, with enough focus, you will catch the ambulance.

The question is: …what then?

I've heard this attributed to Warren Buffet, but I couldn't verify that.

Popularly attributed to Picasso.

Most people see only one ply.

When a mult-ply peron sees how the first ply lines up with later plys they get extremely excited in a way people who only see the first ply get confused by "this looks basically like the other one… what am I missing?"

Also, a great one ply idea that runs into a wall on the second ply they just can't even pretend to be excited by.

So people think they're not being a team player, because the idea is great on a single ply but bad in a way most people can't see.

The multiply thinkers are sensing a dimension that other people can't see.

Disappear, teleport, reappear.

Radagast is a fox.

You find yourself in a massive, fertile valley stretching out in front of you.

It's glorious... and yet it's still overwhelming.

Which path do you take first of all of the choices in front of you?

And how long until the others find it, too?