Bits and Bobs 2/10/25

System 1 - Automatic, cheap, parallel retrieval of past experiences.

System 2 - Effortful, expensive, serial creation of novel reasoning.

It looked like they were able to reason, but that was due less to them being capable of reasoning and more due to their planetary scale.

If you thought of them as just like a normal single human, you'd miss the fact that they are a totally different category: a planetary-scale hive mind of vibes and memories.

LLMs can quickly retrieve past vibes and tweak them to fit a novel pattern, but only a little bit.

But with planetary scale, it doesn't really matter–there is such huge coverage of scenarios that there's likely a pretty good one to draft off of.

As a result, we tricked ourselves into thinking they could do novel reasoning when really what they were doing is cache retrieval with a bit of augmentation.

This is why we got results like this one showing that the previous class of models couldn't do compositional tasks^[aab]–its faux reasoning wasn't powerful enough.

The System 1 has enough coverage that if you give it the time and space to think step by step, it can do proper reasoning even for truly novel scenarios not captured directly in the System 1.

It has humans-style common knowledge it can use to reason its way through novel scenarios.

These models also give a reinforcement learning style curve of quality for that reasoning ability to then climb up.

Turns out that computers need time to reflect and think to get better answers, just like humans do!

The computer System 1 is orders of magnitude beyond any human's System 1 that has lived or could ever live.

The computer System 2^[aad] is currently high school graduate level.

"In all, I found that using Operator was usually more trouble than it was worth. Most of what it did for me I could have done faster myself, with fewer headaches. Even when it worked, it asked for so many confirmations and reassurances before acting that I felt less like I had a virtual assistant and more like I was supervising the world's most insecure intern."

What if it decides you're in its way?^[aal]

Optimized for efficiency for the market as a whole, not my individual nutrition.

Nobody could tell you no on your own computer.

You could install whatever software you wanted, and combine data from applications in novel ways via the filesystem.

If you didn't like how a piece of software worked, you couldn't tweak it or configure it to connect other tools on the data in the filesystem.

You can use it or not use it, that's the only two options.

A thing that people don't actively hate that's minimally better than alternatives.

But it takes time for the tree to start growing and become big and strong.

If you need something big and strong right now, planting a tree will never be the thing you do.

But if you had planted a tree before you needed it, you'd have your own self-growing thing.

Linear process here means "An employee has to invest effort to get it to handle the increased scope of the ecosystem."

A directory like Yahoo was would be an example of a linear process.

In the product we were building we needed to know which attributes were actually synonyms (e.g. "date of birth" and "birthdate").

There were only a few dozen equivalence classes that covered almost all use cases, so I wrote up a CSV of configuration and we tried to check it into version control.

A search veteran refused it: "You must check in the process to generate that list, not the list itself."^{[abf][abg][abh]}

That meant that as the ecosystem grew^[abi]–as more users used it for more attribute names we'd never thought of, or in different languages, even without human intervention the system would be able to adapt and heal, automatically.

This iron rule was one reason that the search engine could coevolve with the underlying, open-ended ecosystem.

A tiny bit of additional work now made the system antifragile and auto-catalyzing, a compounding process, not a linear one^[abj].

At the beginning they work great.

But they are fundamentally capped at some frustratingly low ceiling.

You can always get more return for more effort, but past a certain point it's effectively meaningless, all effort no return.^[abk]

This is a vicious spiral.

Close-ended, top-down contexts have this shape.

Think of the portion of real world value captured by the system as the volume, and the set of rules and structure necessary to capture that value as the surface area.

The real world is fundamentally, inescapably wrinkly and fractal; it does not behave according to simple linear rules that make sense to humans.

As you get into smaller details, they have more surface area for a smaller amount of net new volume.

This is where the shape of the curve comes from.

The volume grows logarithmically and the surface area grows exponentially.

This scenario cannot be tackled by simply putting more effort into it; the curves race away from each other.

This problem only shows up if you try to capture the real world with top down rules.

At the very early stages, you get a high amount of value for a small amount of effort, and everything seems to be going great.

But unbeknownst to you, you are locked into a path of tragedy and heartache.

As you get further, you get closer and closer to the inflection point where the curves cross.

When the curves cross, you start getting extreme cost for very little benefit.

As you get closer to the inflection point, you can tell something is wrong, but you think you're just losing your touch, and you push harder.

Once you get past the inflection point you are lost. You are locked into this approach that used to work but now cannot work.

Starting from scratch seems impossible, and so you toil there, stuck in a dead end, increasingly exhausted and angry, until you give up.

You can get to 80% of the value with 20% of the effort, but you'll need infinite effort to get anywhere meaningfully beyond 80%.

This showed up for Alexa and Google Assistant: grammar based trees of behavior that got increasingly expensive to author for real world scenarios.

This shows up any time you try to capture a real world phenomena in a formal ontology.^[abl]

This shows up in a government trying to set precise laws.

This is also part of why you get the tyranny of the rocket equation for organizations.

It's also why organizations and organisms^[abn] get to a certain size and can't get bigger whereas emergent things like an ecosystem or economy or city can grow to an arbitrary scale.

These combine the two warring curves in their maximal expression.

This is the virtuous cycle.

Open-ended, emergent contexts have this shape.

This shape is wildly different from its cousin.

It uses an emergent phenomena to capture an emergent phenomena.

These kinds of situations are rare and hard to engineer, but they can be searched for and grown.

To solve modernity's problems, we need to focus more of our attention on this kind of virtuous cycle.^[abp]

It can only be tackled with swarm energy that is emergent and imprecise.

A classic network effect.

Even with billions of dollars of investment, it's sometimes not possible to get to the critical mass point.

You need:

1) a core that is valuable to a subset of the market on its own

2) a boundary gradient that pulls in more and more people over time.

The boundary gradient can be a network effect, or it can be as simple as a thing that is values aligned, that people want to want.

If a single component fails the whole thing doesn't work.

This means all of the focus is on improving the weakest link.

Only the best link matters; everything else can be ignored.

This is one of the characteristics of antifragile systems.

Science, evolution, capitalism, etc.

Everyone already agreed it was important, so there's no pop of innovation value.

The competitor with the fastest OODA loop will win, all else equal.

As predation happens it kicks off an accelerating arms race.

The pocket has a different centroid than the main context, which might by happenstance be in a direction that is innovative.

Though note most "innovations" they find aren't viable in the main context, it's only the small set that make it through the structural percolation gauntlet crossing over to the main context that work.

The gauntlet is the selection pressure that culls the emergent process's outputs.

But that only works if you have a random sample in the pocket.

If there is a structural selection bias (which there must be in an emergent pocket, at least in some dimension) then it ceases to be a good proxy for the surrounding context.

Aligned: "need" + "want to want". Misaligned: "want" - Not motivated to do what you know you need to do.

Aligned: "need" + "want". Misaligned: "want to want" - A gay person not yet out to themselves.

Aligned: "want" + "want to want". Misaligned: "need" - Need personal growth and development to understand what you need.

Kayfabe is a carny word that means "a thing that everyone knows is fake but everyone acts like is real."^[abv]

It's often applied to professional wrestling.

But it also applies to politics within organizations.

But over time, the social processes necessary to work together inside the organization towards a coherent outcome metastasize and take over the organization's soul.

Imagine if in every team meeting when the boss proposed a new goal someone raised their hand and said "here's ten reasons I think this will definitely not work."

In that case, the plan definitely doesn't work, because no one on the team will even try.

But if everyone entertains the idea that it might work, maybe as we work together we find a way to make it work.

One of your projects is objectively in a "yellow" state, but by the time the final roll-up is presented to the CEO next week, you'll have gotten it to a "green" state.

If you mark it a "yellow" now, you'll be more likely to attract scrutiny that could randomize you and create extra meta-work, and it will be solved by the time of the report anyway.

What do you mark it down as? Yellow or green?

What most people would do is mark it green.

Each person has an incentive to greenshift^[abx][aby] just a little.

But if you greenshift on top of a thing that's already been greenshifted, the greenshifting multiplies together.

That means that up multiple ply it could be off from the ground truth by many orders of magnitude.

If you let go of the kayfabe, you'll get stabbed.

If you let go of the ground truth, the outcome won't happen… but the social complexity makes it extremely indirect to attribute outcomes to actions anyway so you'll likely be safe as long as everyone thinks you're working hard.

So the kayfabe tends to ratchet up and up.

Once you have let go of the ground truth, all you have is the kayfabe, and defending the kayfabe becomes the end.

If someone threatens it, they are threatening your infinity, and they must be stopped no matter what.

So you stab them.

To ring the ground truth bell knowing you will get killed for it.

To stab the person who is about to ring the bell.