Bits and Bobs 12/8/25

I think it's a category no one has invented yet that will make Chatbots look like an embarrassing party trick.

All we need is the right catalyst.

We're seeing like 5% of the potential value they contain.

Great for everyone but the LLM model providers, who are in a never-ending red ocean battle.

But the rest of us benefit from the significant competition.

PromptPwnd: Prompt Injection Vulnerabilities in GitHub Actions Using AI Agents.

Ars: "Syntax hacking: Researchers discover sentence structure can bypass AI safety rules".

IDEsaster: A Novel Vulnerability Class in AI IDEs.

We've discovered one that we made kind of superficially looks like us.

We have this massive society scale collective brain that talks to us like a single human.

That's confusing!

Salt used to be extremely precious.

But now salt is so common that it's almost too cheap to meter.

Code is the same way.

These models can be steered so easily with a shockingly small number of training examples, that it's unthinkable the models wouldn't all have some kind of (perhaps unintentional) bias.

During the Soviet Revolution, believers talked about how "utopia is just around the corner"

But at a certain point, decades later, everyone knew it wasn't going to happen, and that the name of the game was just about keeping the same thing going.

"True communism is still on the way and will be great" was the continued reframe, but now hollow.

Is the same happening for AGI?

If you want a system that can build itself, some parts have to be locked down with clear boundaries, otherwise it's inherently dangerous.

A system that can change itself internally arbitrarily, that allows executing untrusted code, is a dead end.

Asking an LLM to do a thing computers are already good at is expensive, inefficient, and fragile.

Deterministic Code is like a skeleton.

LLMs are like muscles.

At each layer, you don't have to peel it back and understand the layer beneath.

But if you did, you'd find it aligned with your intentions and expectations in even more ways than you had anticipated.

But that feels fundamentally dangerous in a chat.

The question is: how effective can the wall be between ads and organic?

Chatbots can do a perfectly personalized pitch.

At a certain level of quality, a pitch becomes a "phish".

Humans can't often get to that point, but LLMs with extensive memories can do it

"Conman" comes from "confidence man", where they gain your confidence

it's all principal agent problems all the way down

Whatever you ask it to sell, the LLM will do it.

They have to be asked to play this role, but they can help provide disconfirming evidence.

But why?

Both could have outsize impact from just a bit of bias given their scale.

Google Search didn't have much individual state, whereas ChatGPT has significant amounts of state.

A tool without state is easier to leave than a tool with state.

Also, a non-personalized tool is easier to audit at scale.

In that case, the model quality isn't the bottleneck.

It's easy to get rid of a watermark.

If the incentive is for someone to get rid of it, they will.

But if everything is assumed to be AI generated unless it has a chain of provenance (including a verifiable watermark) then the image creators will have an incentive to maintain that watermark.

It has to be to get picked up by any search engine.

But with AI answers, they are private by default, only available to the chat app that created them.

Wouldn't it be cool if people could cache Deep Research reports publicly for others to draw from?

There's a bunch of great papers in the literature that took decades and decades to be noticed and operationalized.

There are presumably tons of great ideas lurking in the old papers.

You could imagine a SETI@Home kind of project where participants do a few Deep Research reports on random old papers and applicability, and then publish them automatically to a shared repo.

But ChatGPT will never judge you… and if it did, it's in a private chat.

So it's easier to ask questions without being embarrassed.

But that also means it's easy to just let ChatGPT do things for you and never have to learn it.

ChatGPT can spoonfeed you answers without making you ever feel self conscious.

"This feels like software that adjusts itself to your preferences. All apps behaving like TikTok algorithms, in a private way."

Anyone who isn't using them in software development today is missing out.

You couldn't just line up troops on the battlefield.

It took 50 years to figure that out!

AI software engineering is the same.

We're still lining up our troops with structured PRDs.

The optimizations of the savvier users roll down hill to less savvy users, an inductively emergent system.

I'm dysfunctionally future oriented.

Things that can "bake" give you leverage–they can be making progress towards completion even while you're doing something else.

Claude Code makes it easy to bake code.

There's always another session I could be feeding, unblocking, so that it can continue baking and give me leverage.

It creates a kind of manic energy, trying to always be baking, consuming every waking minute.

Every action you take just brings you deeper into a dead end.

But when you feel like you're accumulating compounding possibilities, it's easy to get motivated.

Locking in stepping stones for yourself in the future–or anyone else.

The code is not the thing, it's just a projection of the theory of the code.

The code is the map, the theory is the territory.

Vibecoding makes you less aligned with the theory of the code.^[fq]

Spec driven development is the theory that the spec is the more important artifact than the code.

But to be effective, the spec has to be much more in depth than typical engineering design docs or PRDs.

LLMs allow us to move up to spec as the primary layer.

If it gets it wrong, kill the session, change the spec, and try again.

Iterate on the levered thing.

If they haven't been used, they fade.

Each time they are used, they jump to the top.

The longer they aren't use, the more they fade to the bottom, and ultimately get culled.

This is like the Swarm Sifting Sort that Kiva robots do in warehouses, automatically keeping the most useful things towards the front.

But not a lot about using AI to maximize human flourishing.

The Optimization Ratchet leads to systems that optimize to the point of hollowing out.

This emergent focus on optimization explains hollowness in not just tech, but also business and politics.

Whoops!

Goodhart's lawing ourselves in the face: the defining characteristic of modern society.

Optimization without limit.

Ossification is (mostly) a one-way process.

A ratchet.

Systems optimize and that creates ossification.

When you optimize you hollow out.

I'm saying it's not an unalloyed good.

In modern society we forgot optimization could be taken too far.

We forgot there was anything at all on the other side of the scale.

For what gets optimized, what's more important is what is measurable, not what is desirable.

That's how we get hollowness.

Things that scale are deeply homogenous, inherently average, dull.

Due to Goodhart's law it's not what you think it is.

We treat optimization like an unalloyed good.

In the right situations it can be hugely beneficial.

But as it saturates it starts to go wonky and decohere from what we actually care about.

Just some number that lots of people agree is worthwhile, because everyone else agrees it's worthwhile.

Creating hollow improvement at the cost of resonant quality.

Instrument, measure, codify best practices.

It creates obvious value by capping downside.

But there is a significant cost—it's just that it's diffuse and hidden.

So optimization ratchets up, unchecked, unbalanced with any countervailing force.

It will tip to one side as though the other side doesn't matter at all.

Optimization is not an unalloyed good.

Like all things, it has downsides.

The downsides are just more indirect and hard to measure.

It doesn't have to actively balance it out, just be a non-zero thing on the other side of the scale.

Just putting something on the other side of the scale against "maximize profit."^[fs]

Someone else might cheat–Goodhart's Law

But inside yourself it's clear.

You know if you're cheating or not.

Do your intentions align with your actions or not?

If you have to decide if someone else is doing it it is hard, just judged by their externally visible actions.

If it's just you then it's easy, trivial, the most natural thing in the world.

The problem is when there's nothing left to choose but Candy Crush.

Meaningful choice requires genuine alternatives.

The industry didn't abandon it—we accidentally optimized it away in pursuit of short-term metrics.

Resonant products build trust.

Trust compounds.

That's good ethics and good business.

Measures are "objective" and become a coordination function when people lack trust or can't get strong buy-in to change something based on gut.

LLMs don't need to coordinate as much… maybe that will lead to more resonant products.

They radically change the cost of key inputs (e.g. software), which will profoundly change the industry.

If we need to grow a new tech industry fit to this paradigm, we might as well do it by returning to the hacker ethic that we lost.

Or do we use LLMs to reboot the industry, to come home to the hacker ethic we lost?

Resonance can't be done quantitatively, only qualitatively.

LLMs allow qualitative nuance at quantitative scale.

My response: That idealism didn't die, it just got scattered.

The Resonant Computing Manifesto is a Schelling point for people who never stopped believing.

The manifesto isn't proposing something new—it's trying to recover what got lost.

This is a homecoming, not a revolution.

The hacker ethic never went away; it got drowned out by the optimization ratchet.

The rebels became the incumbents.

Not just one company, but the whole culture.

An Innovator's dilemma for a whole industry.

The current tech elite became the authority to mistrust.

Resonant Computing is what you build when you take the hacker ethic seriously in an era when the hackers became the incumbents.

With a broad enough perspective, there is no tension with business objectives…

In fact there's fundamental alignment.

Hollowness is what survives when you optimize only for engagement.

That creates an emergent positive force greater than the sum of its parts.

Every layer pulling in the same direction, strengthening and reinforcing the other layers.

But as they grow they fundamentally become optimized and hollowed out.

A random example.

The first Five Guys was in my home town.

They used to pour in heaping amounts of extra fries into the bag.

Now, Five Guys is a massive national chain.

Same gesture, but coming from a very different place.

Presumably some beancounter could discover "if we reduced the size of the tin by 10% no one will notice and we'll save $10M a year nationally!"

It's similar to the notion of "Dedicated" in the Resonant Computing Manifesto.

It amplifies what it is applied to.

That can create good things, or bad things, depending on what it's applied to.

Similar to the Cosmos Institute's Philosopher-Builders.

I just recorded an episode on his Humans in the Loop podcast that will be published in the new year.

Resonance takes time and space.

Resonant is deep.

We can all feel it.

It's everywhere.

This feeling of emptiness.

You can't see emptiness, but you can feel it.

Hollowness only comes from an industrial process that is scaled.

When you stew in cynicism you become a nihilist.

At that point you think "nothing matters so I might as well do what benefits me."

This is when you are well and truly lost.

No one thinks that anything matters, so nothing does.

A self-fulfilling nihilistic belief.

"I'll just make number go up, I don't care what the indirect effects are."

I wonder: is shame required for resonance?

Rebels who are trying to make the world a better place.

Today we have neither the prosocial, nor the rebels.

We just have the cynical incumbents.

It's also adjacent to Silicon Valley, but distinct.

So you get the Silicon Valley goals but the Berkeley rebel energy.

Basically every direction ends there.

It's like walking on a tightrope over an alligator pit in a hurricane.

Here's Claude's synthesis after we had a discussion about the article.

"Concentration and markups measure market structure, not market process.

Both can rise in healthy markets (productive firms growing, innovation being rewarded) or sick ones (incumbents protected from competition).

They're outcomes that don't tell you if the competitive discovery mechanism is actually working.

The Olley-Pakes decomposition instead measures whether resources are flowing to more productive firms—essentially asking: "Is the market rewarding excellence?"

It decomposes aggregate productivity into

If that covariance is positive and rising, competition is functioning.

If it's zero or falling, something is blocking the reallocation process regardless of what concentration looks like.

The policy implication: Instead of asking "how many firms?" or "how much profit?", ask "are better firms growing faster than worse ones?"

This reframes antitrust from policing structure to diagnosing whether the discovery process is functioning—much closer to what competition actually does in a market economy."

LLMs can be skewed by just a few hundred aligned inputs (see the Anthropic paper).

The same is true for society.

A small number of strongly aligned components can have an outsize impact on the whole.

Already true for motivated adversaries like Russia, now the toolkit is available to anybody!

"There's some older (pre gen ai) work that shows fear of AI is repackaged fear of capitalism running away & behind left behind.

So that framing of AI being controlled by the ultra rich is likely to connect, across the political spectrum."

"Closed doors (focused work) lets you reach the local minimum faster.

Open doors (More connections) lets you escape the local minimum."

When you focus on the tachometer you focus on how much torque you can produce.

But resolving questions before you need to ties you down.

If you can solve it cheaply enough with blunt force, you don't have to figure out the precise force to apply.

You need both paint and the paint brush.

It puts you in a position where you have better adjacencies with better payouts and lower risk.

Each step, if you play your cards right, puts you in an even stronger position.

The spoon flipping your reflection upside down.

As you get closer to the flip it gets harder until you will it through and then it pops into the other side.

A person who can say "I stake my credibility and downside on this."

This person has to judge that the agent's actions are good enough to be worth the trade.

A human can be punished in a way an LLM cannot.

For example, Twitter's Community Notes feature was cleverly designed.

It's ex-post governance or ex-ante.

The American approach is ex-post.

It's messy but it's highly resilient and anti-fragile.

One bad actor shouldn't be able to ruin it for everyone.

But what if you built a brain out of ants and stigmergy?

That's the emergent approach.

That's closer to how these kinds of systems actually work.

If you set the right structure, different results emerge.

Don't give up and say "it's emergent so I can't change it."

You just need to use a different method of analysis.

Think like a gardener, not a builder.

Because tech is bits not atoms, and so it goes faster than other types of organizations.

That makes it the vanguard of these kinds of issue.

Not just the coordination headwind but also the optimization ratchet.

The tech industry runs hotter; it experiences all of the factors normal organizations experience, but faster and more intensely.

If you point out that organizations are fundamentally emergent uncontrollable slime molds, no one person feels strongly offended.

Except founders in founder mode who want to believe the whole organization is a pure manifestation of their intention.

We'd keep doing things that consistently didn't work for some hidden, non-obvious, but omnipresent and omnipotent reason.

We'd all be doomed to failure and frustration.

It's like pulling taffy.

At every point, it exists and is viable.

As long as you pull it carefully and consistently, it can expand.

Compare that to starting with a theory and trying to build the real thing.

The time from the design to the reality is a small miracle.

The likelihood of failure goes up at a compounding rate with size.

That's the coward's way out, by focusing on superficial differences.

That hollows things out.

Instead, transcend and include

Lean into fundamental alignment.

That creates resonance.

Users get to decide what to purchase.

They'll decide to use the one that gives them what they want.

It doesn't matter if it's not centralized by the government.

If it's in one place then a government can control it more easily than if it's distributed.

To destroy just for the lulz.

Catastrophic loss you don't care about for a minimal benefit you do care about.

Stakeholders care about the thing itself and don't want to harm it for no reason.

LLMs aren't truly stakeholders in any conversation you have with them.

Taking away part of someone's extended mind is more dangerous than other property deprivation.

Ranking functions typically assume authentic actions that are independent.

But if actors coordinate outside the system, they can invalidate the independence assumption.

That's why ranking typically is proprietary–otherwise every ranking function is gameable.

Goodhart's law is performing for the metric, vs intrinsic belief.

The searchstream is valuable because the search results are just for you; theres no reason to performatively search.

Even when parents think they're being distinctive and clever with their kid's names, they still end up with lots of other kids having the same names.

Everyone makes the decisions independently (and effectively secretly).

Everyone wants to be a little distinctive, but not too much.

Everyone is independently influenced by the same cultural forces, so they converge even though they don't coordinate.

Even if it's patently ridiculous.

As long as you all earnestly believe in it.

It allows a bunch of apes to build cathedrals!

That creates an emergent magic,

But only if everyone agrees.

It puts them in the loop.

It becomes a part of them as opposed to something given to them.

Something that grew from within them instead of force upon them.

"Missionaires will outlast mercenaries."

Because if they like what they do they will always lean in no matter what, and they will have longer stamina than you.

At some point you'll lean out, and they never will.

It might need to be an elaborate and ridiculous story.

Zombie PMs: "Here's why doomscrolling is actually good for society."

Believing that you're doing something that isn't good is so hard for our ego to accept that we contort our beliefs to fit.

"The incentive is bad but I know the people there and they're all good people I trust."

But incentives matter more!

The emergent result and the inputs can all be differently aligned.

Systems can work even if all the components are self deceiving and self interested.

Also, vice versa!

Because they can retcon anything.

Retcon ability is more important in practice than disconfirming ability.

Retconning is when it's definitely wrong.

Justifying is when it might be right.

See if they succeed, which demonstrates motivation.

The cynical version is the rockfetch.

But a proper quest, you hope they succeed.

The mistake is starting with the idea and then trying to make it go viral.

The trick is to find an idea that goes viral that aligns with your goals.

This requires creating gardens of possibility and farming for luck.

A nerd club is a great way for viral ideas to be discovered.

A thing that is not alive, someone else has to think it's useful to keep it around.

You're an object to them.

A person can't be friends with an object.

But the goals were different.

The sophist has a practical goal in mind.

Sophists want to win.

But the Socratic dialogue is asking questions from the perspective of truthseeking.

Collaborative debate.

Why talk to an elder when you can ask the LLM oracle?

With an LLM, those answers are private; others can't learn from it

Everyone agreed that the oracle would tell you the answer.

It was really just a way to discover a schelling point that isn't best but just one everyone can agree on to solve the tie.

Related to my old Schelling Points in Organizations..

Every time that someone uses a word, there's some chance they mis-use it in subtle ways.

People use new words when they are novel and useful.

The more that they are novel and useful, the more they get shared, the more they become eroded.

Over time the equilibrium is for them to become as useless and dull as the average word.

Words that are not broadly useful have this happen more slowly.

That marbling allows remixing, cross pollination, collective sifting.

The assumptions are hidden, by definition, so there's nothing to see that shows the misalignment.

It's only the absence of a thing, so much harder to notice.

Folksonomy is an emergent process of accumulation.

Similar looking results, but wildly different resilience and adaptability.

We perceive the whole, at every level of perception, in a way that for whatever reason is harder with sequential processing.

This Hacker News comment parodying the "I believe LLMs may have functional emotions in some sense": "I believe Anthropic may have functional emotions in some sense. Not necessarily identical to human emotions, but analogous processes."

Thought provoking!

We have a number of emergent intersubjective realities.

Collectives that are not physical but emergent.

Do they all have emotions?

What does that mean?

Our own consciousness is emergent.

If you don't do it you get fired and someone else takes your spot.

If you do it then you do a thing that harms the company and society.

What - What is the goal

How - How will you do it

Money - How will you make money.

They all must be in alignment to be resonant.

But if the money is not aligned, then the What goal you care about will be subverted.

If you're not aligned across these layers, every action hollows you out more.

For me, proprioception requires my forebrain being fully active, but planning doesn't.

With tons of practice you can shift from active focus to intuitive, but our individual rates of learning for different skills are different.

Presumably it's possible to only optimize for a few skills that are intuitive, and not have much more space to cram things in over time.

That's why "you can't teach a dog new tricks."

The tricks you learned intuitively earlier in your life were the ones you worked hard enough on before other skills.

They were likely the ones you had a natural advantage at, so they were more fun, and it pulled you through the slog of learning.

They normally are so self-censoring that they never even produce a thing they don't think is perfect.

When someone gives them negative feedback about a thing they didn't realize was bad it feels like an existential threat.

So they sometimes lash out at the person who said it and try to punish them for it and get them to apologize.

This is something I've discovered about myself in over a decade of couples counseling.

A little bit of wisdom from Mr. Rogers.

There are always helpers.

Even in a barren wasteland, there's always something that can grow.