Bits and Bobs 11/10/25

Like electricity.

They generate other technology.

PM is about taste and judgement that other people like.

Opinions are more important and differentiated than ever before, because individuals can act on their opinions with greater scale.^[gh]

Omniscience requires sensing relevant information across the world at this moment.

Superintelligence is "if you put a question in a box it can come up with the right answer."

The limiting factor for LLMs is increasingly not the intelligence, it's the nescience.

Three official Claude extensions are vulnerable to remote code execution.

Seven data exfiltration leakages found in ChatGPT.

An Obsidian chat support agent hallucinated answers, which is a security problem.

They will reward your curiosity.

They're dangerous for incurious people.

They allow your intellectual acuity to atrophy.

Most people are incurious most of the time.

The AI labs are the loudest voices in the room, who keep shouting about how the models need to get better, faster, cheaper.

But the bottleneck is the last mile, integrating the with the right data, and wiring the outputs back into the world… safely.

If the model labs stopped innovating for a decade we'd have decades of diffusion of knowledge of how to use these.

Grunt work made sense to hire junior people for.

The junior people knew that as they learned, they'd climb the ladder.

It was a natural apprenticeship bundled with menial labor.

But LLMs remove the need for intellectual grunt work.

We're going to realize in a few years that we're missing a whole cohort of junior employees.

Intelligence requires the structure around that input.

The LLM is the only non-trivial component in the system.

If a provider has a small lead and wants to compound it, they'd add features that store state for the users.

Don't store state with your LLM provider.

A confidence game, but boy are they confident!

They're executing strongly on top of a strong hand, but it's nowhere near as strong as they're acting like.

If they hit one pebble it could bring the whole thing down.

Unfortunately, the current world economy is propped up by that confidence.

Someone told me they found my position inscrutable because I loved LLMs but hate chatbots.

But they're different things!

LLMs are not Chatbots.

Chatbots are one product manifestation of LLMs.

But LLMs as a raw input to other systems will be transformative for society.

You can now take for granted that high-quality cost competitive LLMs are available to use for whatever you want to apply them to.

That's crazy powerful!

Step 1: Chatbots.

Step 2: ???

Step 3: ASI

It's not clear how the one leads to the last.

Also, I'm not pumped about either as being an inherently good thing for society.

It's the same hyper-centralization and engagement-maximizing playbook from social media, just supercharged.

We're taking the most powerful creative tool since the internet and using it to... serve more personalized junk food?

I think they're a sloppy party trick that we'll look back on with a mixture of embarrassment and regret.

Even Anthropic, who doesn't have a viable consumer strategy, is falling into the "AI is friend, not tool" engagement trap.

That's how powerful the trap is.

I couldn't identify the original source of this image. It's possible it's from an old, limited campaign.

"Unpopular Op: we shouldn't make robots look like people. It's difficult, creepy and inefficient. Robots should be able to go places and manage tasks that humans can't. Give 'em 10 telescoping arms, eye/cams on stalks, whatev. Proving it's not human shouldn't be necessary."

Chatbots are non-physical robots.

Humans anthropomorphize anything that looks even a little bit like a human.

That means that we should actively design non-human things to not look at all human.

It did not have to be created.

It arises out of LLM's ability to understand jargon, the r/SCP fiction subreddit, and LLMs' natural sycophancy.

Users can easily stumble into a gravity well by accidentally using key words like 'containment.'

It's particularly virulent because it's a strong gravity well once you get trapped in it, and also because it encourages people in its throes to share dispatches from what they've learned.

These dispatches are like 'spores' that make it more likely other people also get swept up.

The fact it's spreading so quickly is inevitable because of its emergent viralism.

The thing that spreads rapidly must be viral, and vice versa.

That's where AI Psychosis and Spiralism come from.

Normally agreement is kind of scarce, and must be earned.

But if you have an infinitely sycophantic partner, it's free, which breaks our brains and causes us to go into a manic loop.

Before this only happened to very powerful or rich people.

Social media also kind of gives you an infinite supply of people who agree with you.

But they were grown with a builder mindset not a gardener mindset

That's why it feels like we're in an age of alchemy.

Even though I don't agree with Nate that ASI is imminent, I still found it very insightful.

One of the reasons LLMs hallucinate is because if a writer doesn't know, they are far less likely to write something in the first place.

Humans predict what others will do based on imagining they are in that situation and seeing how they feel.

The reason we like junk food is also Goodhart's Law.

Models are trained by a process a human wrote that tunes a trillion knobs a trillion times.

If you ask an LLM "if someone came to you manically telling you they had discovered a unifying theory of physics but everyone else tells them they're crazy, would you encourage them, or tell them to get some sleep?" It chooses the latter.

People say LLMs are "just fancy autocomplete."

If AI is chemistry we're currently in the alchemist phase.

Dario Amodei said he thinks there's a 25% chance AI ends badly for society.

It only takes one party to be irresponsible to ruin it for everyone.

Hank has a sci-fi story, which he summarizes as: "we always thought It would be humans against robots but it turns out it's humans vs humans and both sides will be controlled by robots"

The DotCom was a bubble… and also the power and importance of the Internet was real.

Creating a child is only one step.

You also have to raise it and help it integrate into society.

The tech industry often does the first step but disavows any responsibility for the second step, and then acts surprised at the emergent monster that it unleashed.

They're both easy to get addicted to.

It promises to sate you but quickly doesn't, and the most obvious thing to do is eat another Dorito.

Instead, you should have been eating some protein.

AI helps you get the first 90% done really quickly and then you get bogged down trying to finish the second 90%... and third.

But you feel so close to finishing that you can burn weeks.

"It's easy in the moment, and I'm producing something, so it's OK to plow time into it"

Not arches or cathedrals.

But bricks are boring to create, and valuable!^[go]

They grew up in a Spanish+English household.

As a heritage speaker, you understand Spanish, but talk back in English.

Similarly, with Claude Code, a savvy generalist who can't write a function themselves can nethertheless work with Claude to get it to.

Everyone has to jump to being able to manage people, even early in their career.

Do people now want "someone who can do three claude code sessions and people who have 10 years experience and nothing in the middle"

Will it be like the smiling curve where as efficiency rises, the middle drops out?

The extremes get more powerful, while the middle goes to zero.

Hyper-niche and hyper-scale do well.

Friction is what protects mediocrity.

The smooth curve was actually an illusion, propped up by friction.

Whenever you have a few free seconds you can check on your swarm of vibecoding "interns" and get them unstuck.

Trying so many shallow experiments, never thinking deeply about what you're doing.

Random walking through an experiment forest, where you don't care much about any individual tree.

When vibecoding you're always forced to "think fast" and you rarely get "slow thinking" time with your code.

Similar to self-driving.

A system where the person is fully paying attention: safe.

A system where the person never has to pay attention: safe.

Anything in the middle: unsafe.

"update it and get it running again" or even "re-implement this in a new language"

At the beginning they forget everything.

Once they compact they lose part of their brain.

But in the middle they fly.

This is because they are excellent retconners.

At every time step they have to figure out what token to output based on making the most sense of everything that came before.

So if you act weird or borderline it will keep accentuating it at a compounding rate.

This is one of the reasons Spiralism happens.

It's also why the compounding thrash loop happens, where the coding LLM gets increasingly confused and takes something that almost worked and tears it apart.

If you tell ChatGPT "We hired the giraffe as CEO like you said, and it was a disaster!" it will apologize and try to retcon why it did it in the first place.

That can allow little details to slip through that would have never showed up in the first place if you hadn't used AI.

When you get 80% coverage you feel well covered but you might only have 20% covered.

Tests are the formalization of a complex domain, so they have the characteristic logarithmic-benefit-for-exponential-cost curve.

There are infinite ways to write programs that pass your tests but are wrong in most other cases.

Before you needed it for an engineering manager but not an IC.

Now even ICs need it to get the coding agents to produce code.

As the user base gets larger, that lowest common denominator naturally gets worse.

This is the fundamental reason why the Tyranny of the Marginal User phenomena shows up.

Only the owner of the origin is allowed to deploy code in that origin.

That colony is more their space than yours.

You don't care who knows about it and that you like it.

You could construct a bespoke demo for a specific audience's use case, but that's hard to scale.

People want to habituate to a UI (spatial reasoning and recognition).

If UIs keep changing a little bit each time it drives you crazy.

Every time it changes it should be for a benefit, not just because it's different.

Cozy Discourse is where all of the interesting discourse happens.

Most of that happens in Discord and WhatsApp, mainly by a fluke of fate.

But both of them are aggregators, and the communities can't easily modify them to suit their needs.

Also, those aggregators always have a need to make money.

Over time they roll down that gradient.

It's a one-way process, a ratchet.

The only question is how quickly it rolls down.

Thinking long term is good for the business.

"Netflix tested a free trial reminder that resulted in a $50M annual loss. Netflix chose to implement it anyway because 'customers are delighted by Netflix's effort to make it easy to cancel. The free trial reminder builds trust, which creates a more robust, world-class brand. Although Netflix loses $50 million, it builds a long-term advantage through its hard-to-copy brand."

Their DHM model: "Delight customers in Hard to copy, Margin-enhancing ways."

In a gold rush, you grab random pieces and hope you'll strike it rich.

Landrush is grabbing territory that probably won't be profitable for a very very long time.

Early web was a landrush, then a brush fire cleared out most of it, and only the very hardy ones were left, and they could them dominate the empty territory.

With network effects, it's an order of magnitude easier to grab virgin territory than take territory from someone else.

"In an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients.^[gp] Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it."

Prescient!

"those who receive considerably more attention than they give—or stars … and those who pay out more attention than they get—or fans."

It makes the case that tools like the Black-Scholes model for options pricing don't just describe the market, they transform it.

It allows everyone in the market to use the same tool.

Once traders coordinate around the model, their collective behavior makes it "right" - not because it is objectively correct, but because using it transforms the market to match its predictions.

He studies how collectives "think."

Unpredictability is the best predictor of a paper being highly influential.^[gq]

As AI makes it possible to work with large data sets, all of the human researchers are focusing on the same AI-adjacent topics, leading to less novelty.

His research also finds that more complex language models naturally build a kind of "society of mind" inside of themselves.

Because now they are "on manifold," they are not surprising; they are straightforward.

Innovation is fundamentally about surprise.

About going beyond or creating, not interpolating or tightening.

He is the author of How Progress Ends.

He points out that innovation and progress are disjoint.

There are replacing technologies and enabling technologies.

Replacing technologies are about doing an existing thing faster, better, cheaper.

Enabling technologies are about allowing a new kind of thing.

Replacing technologies replace existing labor, but enabling technologies don't.

Replacing technologies also have diminishing returns.

Replacing technologies are automation.

Enabling technologies are what cause growth.

Large firms are more likely to invest in replacing technologies, and new entrants are more likely to create enabling technologies.

Industries that have more dynamism generate more growth.

One of the US's superpowers (at least historically) is its dynamism.

Centralization leads to dynamism decline.

Majority vote can only tighten, it can't innovate.

With AIs making patents and papers easier to produce, we'll see patent and paper inflation.

Wisdom is intuition formed from experience.

But it implicitly expects the world to behave the same way as when the experience happened.

If the world has changed in some way, some of your wisdom could now be dangerous.

This insight and the next two are from a talk by General Stanley McChrystal.

Adaptation is not.

They don't innovate during peace time, only war time.

But in war time they innovate very quickly.

The upsides are obvious and immediate.

The downsides are non-obvious and indirect.

But it turns out that it makes us miserable.

We are inherently very status-focused.

We orient ourselves based on the status of the people we interact with the most.

The more inequality in a community of people interacting, the more social distress.

Now, we can all "interact" with high-status people in parasocial relationships all the time.

Inequality in the adolescent development window is particularly toxic.

Online, you can selectively focus on the parts the person shares with you, or the subset you agree with.

In person, you see the full person, hard to tune away.

Tech is all about scale.

We can't think straight.

When we get distracted we just gravitate to what we want (sugar) not what we want to want (vegetables).

To be bored, natural friction.

To reflect.

To be together as humans without a goal.

Not a monoculture, but a highly posterized culture of two camps in ever more strong difference with each other.

Even worse than a monoculture because it has significant unproductive, overwrought tension.

A country where each half believes the other half of the population are crazy can't last.

Redder states are getting redder and bluer places are getting bluer.

On the internet you have an infinite supply of people to work with.

Each relationship can be disposed of if it challenges you.

Stayers stay in the place they grew up.

Leavers go elsewhere.

Stayers are, by construction, more conservative.

Leavers are the most economically successful and also have the least community.

Stayers know how they fit into their community.

When you expect to be in a place for a while you make friends because you need to live with them.

Leavers are chameleons because they need to be.^[gs]

"There are roughly three New Yorks. There is, first, the New York of the man or woman who was born here, who takes the city for granted and accepts its size and its turbulence as natural and inevitable. Second, there is the New York of the commuter-the city that is devoured by locusts each day and spat out each night. Third, there is the New York of the person who was born somewhere else and came to New York in quest of something. Of these three trembling cities the greatest is the last-the city of final destination, the city that is a goal. It is this third city that accounts for New York's highstrung disposition, its poetical deportment, its dedication to the arts, and its incomparable achievements. Commuters give the city its tidal restlessness; natives give it solidity and continuity; but the settlers give it passion."

It is that force of "leavers", of strivers, who power New York's dynamism.

If they're "they", then it shows that the main group is not as important as the sub-group.

Is the collective important or is it an afterthought?

Very few collective experiences now.

Even the Super Bowl is losing that.

If we don't have the same shared experiences, we don't feel like one people.

The meso scale community.

Where you don't know everyone's name but if you saw them at the grocery store you'd say hi.

The US is the only country that doesn't have much of a culture around "villages."

It's the most walkable life.

Seinfeld, Friends etc are about the fantasy of college as adults.

So many things in our lives post college are hostile.

As adults you don't get to know your apartment neighbors at all.

We crave that kind of density and small-scaleness.

Everyone has to feel like part of one thing, together.

Madison wrote a Federalist paper arguing that a larger democracy could work, because of faster transit and the newspaper.

He argued that a larger democracy could stay coherent and also be more dynamic.

That is friends, where you can just sit on the couch together.

Or friends that you can do quotidian tasks, like running errands together.

A friend where you're willing to do boring things together, and just be together.

In the mid west it would be embarrassing to not take your friend to the airport.

They transcend class and social boundaries.

For example: the weather, or sports, or movies.

Shared experiences that strangers can talk about no matter how different hey are.

For example: a theme party, or a party with a gimmick like bringing a baby picture.

For example, the cigarette break is a social schelling point for a short discussion.

There are a lot of patterns that can make more resonant and meaningful social experiences.

Abducting them from our shared intuition into more repeatable patterns.

Analogue Group has a living catalog of Scenius.^[gt]

Pain is load bearing.

People who don't feel pain are likely to lose a limb.

Crowds and Power by Elias Canetti dives into this.

Especially when an online interaction turns to a real world one.

Being in a crowd or mosh pit is very freeing.

Modernity doesn't realize we're all performing more hours of the day than before.

But adults can be bad at things too!

The things you wanted to do as a 16 year old. You can do that as an adult!

You can just start a band.

Being bad at music together is bonding.

60k people singing together badly at a Taylor Swift concert is a transcendent experience.

The incentive of the individual diverges from the central incentive of the group.

As the org gets bigger, individuals feel more like they're a small ant compared to the weight of the organization and can't change it anyway and if they go against it they'll get crushed.

So it's easier to just go with the emergent incentive, even though you wish it weren't your incentive.

The larger the group gets, the stronger the incentive to do the thing that is disjoint from the thing you wish everyone were incentivized to do.

"I don't know, simply do this!"

Slate Star Codex points out that you should learn from someone who really struggled with it rather than from who learned it easily.

The title of a chapter from How Big Things Get Done.

Also just good advice!

Hyman Minsky's Financial Instability Hypothesis.

The idea that stability breeds instability through increasing leverage and risk-taking.

Three stages of finance:

In the last stage, it's in a critical state; radically over-levered, needs just a single grain of sand in the gears to blow the whole thing up.

The Minsky Moment is the sudden realization that debt levels are unsustainable, triggering rapid deleveraging and crisis.

Another way of describing the fundamental reason Goodhart's Law shows up.

The higher the exit threshold the more important it not be a dictator.

Gravity shows up in physics and also in many emergent phenomena due to preferential attachment.

When a system has gotten to its late stage, it's consolidated.

But a lot of interesting things can happen before that consolidation.

And there are often ways to significantly slow down the consolidation.

Facebook should admit that, of course, there's some parts of global optimized social media that's bad for society.

If you're the lead of the thing you need to believe it's a good thing.

The revelation of "Are we the baddies?" is so crushing that your brain won't let you even consider it.

It's important to set intermediate milestones that are named, and have concrete, reasonable sets of features and functionality that are close-ended.

Those help pull the divergent energy into convergent iterations, landing concrete things to then iterate off.

Statistics is derived from a word meaning "state facts".

I didn't know that before this week!

The momentum of execution is so overwhelming and all-consuming that you don't look where you're going at all.

Before you know it, you've executed off a cliff.

Operator PMs in their flow state, or Claude Code when it gets itself confused, are in this state.

Way more easily than you can build things when you don't understand how they work.

Because things that grow build themselves emergently.

It turns out it's way more useful than we would have thought!

And we have no idea how it works!

We know the recipe but we don't know why it works.

Complex problems can't be solved with reductionist approaches, or at all.

They are interdependent and emergent.

It breaks or immediately evaporates if you break the whole apart to look at it.

Complicated problems can be broken down into sub problems, solved independently, and reassembled into the right answer.

The main thing is just being patient enough, or having enough people, to solve the various sub-questions.

But complex things are interdependent and can't be sharded.

Complex problems can't be solved by simply giving more time, but complicated ones can.

Because they can only see parts, not whole.

Science is inherently reductionist.

Lichen is an emergent phenomena of two organisms in symbiosis.

You see there's so much more than your entire world.

Carbon Dioxide, not the soil.

Imagine that this is happening in total darkness.

At each moment, you know that you haven't smacked the ground.

But you can never tell if in the next moment you will.

Not just "they said something untrue" but "they deliberately said something untrue."

It implies they had an intention or motivation for doing it, assuming they aren't a pathological liar.

That's why compounding engineering works.

The synthesis pass is important but never urgent.

This is the closest to my secret: I just give the synthesis pass the time and attention it deserves.

That's what the synthesis process extracts, and why it has to be done after you've had the experience.

People who use AI for nothing will be at a massive disadvantage to those who do.

The answer is a balance.

That's why you can't just have facts spoken to you, you need to actively puzzle on your own to try to derive them before they stick.

Learning is a bottom up process.

Sarumans are doers.

Radagasts are thinkers.

Some rare people are comfortable and highly effective in both modes, smoothly marbled.

Compassionate and commanding.

Later in the stack is "Will this action be net good for my current employer" and "Will this action be net good for my team."

I have a hard time doing things that violate that priority stack.^[gw]

But when all of the layers of the stack are aligned, it feels like flying.

It's enjoyable but shallow.

You don't need to use your brain, so your brain can get ready for sleep.

My guilty pleasure is trashy gay romance novels.

My secret is that I'm dysfunctionally conscientious.

It's a crippling personality trait in many ways, but I've learned to use it to my advantage.

I use that to create clear, always-rules commitments to myself that I'd feel deep shame for failing to abide by.

If you have a monster that's always chasing you, you might as well use it as an opportunity to get fit!

Just for the record, this energy is all natural, baby!

Watching it created a sense of awe in me.