Bits and Bobs 1/29/24

Perhaps there are no counter-examples because it's fundamentally not possible for some non-obvious reason.

It's easy to lull yourself into thinking it's possible when you don't see active disconfirming evidence.

A "red ocean" is one with fierce zero sum competition.

A non-red ocean is typically assumed to be "blue"--that is, wide open for positive sum discovery and value creation.

But imagine you come across a vast, valuable looking ocean that is totally empty.

It's tempting to think "wow, everyone else is missing the possibility of this vast blue ocean!"

But if the ocean is well-known and obviously valuable, then if it's empty, it's more likely to be not blue, but dead.

That is, living in this ocean is not viable, in some non-obvious way.

There's some set of perhaps hidden constraints that make it non-viable.

If you look at the cost-per-word to produce the Economist, it likely isn't that different from the cost-per-word of USA Today.

But the quality of the output of the Economist is significantly better than USA Today.

It's tempting to conclude "Wow, quality actually isn't that expensive."

But quality sets a hidden constraint: it's basically impossible to provide at scale.

The structure of The Economist can't produce more than an issue a week.

There's a horizon implied by the constraint of hitting a certain high quality bar.

Beyond that horizon is not blue ocean, but dead ocean.

For whatever reason, quality in that dead ocean is effectively infinitely expensive.

It only looks possible because there are no counter-examples.

But there are no counter-examples because it's not possible.

But it's actually based on external context, especially where partners are involved.

A partnership requires both sides to invest at the same time.

If your counterparty is ready to collaborate now but you aren't, the window of possibility might close.

This is especially true if you have a shy (rarely collaborative) and/or powerful partner.

If they're ready to go now, you have to too!

A muse is like a creative partner; the spark that lights the fire of creation.

You can't postpone a muse; it will simply evaporate.

Just like coordinating with a shy but powerful partner.

Clean things are much less sticky because there's fewer nooks and crannies to get hooked by.

Beautiful things tend to be clean.

But only if they are manufactured/built.

Organic things are beautiful because they are wrinkled and fractal and alive.

Successful enterprise companies do not pitch a product so beautiful and perfect that their customers don't need to tinker with it.

Larger, more savvy customers demand the ability to tinker and customize.

As they tinker, they become even more sticky to the solution.

That is, structures that are kept strong based on tension, and can sometimes even appear to defy gravity.

The fractal dynamic balance of tensions is what makes an org strong and alive.

By being in dynamic tension, you have the opportunity to naturally surf left or right a bit as the context changes the optimal balance point.

The longer you are balanced at a precise point, the more structure that will build up.

This structure helps with efficiency, but also makes change hard.

Too much structure can ossify a balance point, making it no longer a dynamic balance but a static one.

Such a balance point is now "dead"--unable to change, even when the context demands it.

Where possible, it's better to have little micro-adjustments in balance points often, just to make sure you maintain the muscle and maneuvering room to change it in the future.

The precise balance point at a higher level has implications for the balance points of fractally nested other balance points within.

If you balance at too low an altitude, instead of balancing it once and everything naturally tumbling downstream from that, you have to balance it hundreds or perhaps thousands of times independently.

As a reminder, "United States" went from originally being a plural noun (emphasizing the states) to a singular noun (emphasizing the Union) after the civil war.

When you make a figure/ground inversion, the real-world details are the same as before, but the interpretation flips on its head.

This can lead to significantly different outcomes down the line.

Not because it changes anything immediately, but because the interpretation changes your beliefs.

And your beliefs influence what work you will do, and thus how you will evolve and extend the thing in front of you.

In fiction, "plot armor" is where the main hero survives through a series of highly improbable lucky breaks.

This is another example of post-hoc selection of a lottery ticket winner leading to an illusion of causality.

We only bother to tell stories that are interesting, and a story where the hero dies in the first chapter is not interesting.

That means that many fiction stories that we consume are ones where the main character has plot armor, setting our priors.

We are the hero of our own story.

But we're bit players in everyone else's.

Our implicit assumption that we are the hero combines with our priors to lead to a conclusion that we have plot armor.

But this is an illusion!

It must be--you're throwing away nuance to tell a simpler, more clear story.

This happens every time you reduce dimensions or nuance.

But if we didn't do it, it would be impossible to "grab onto" the fractally complex, mushy, multi-dimensional reality.

Caricatures are necessary to get enough clarity to act.

Instead of fighting caricaturization, make sure to pick ones that align up with reality in the dimensions that are most important.

For example, writing a short explanation post to clarify your thinking (if you like writing).

Our intuition is that all things have a positive opportunity cost.

But when the opportunity cost is negative, our intuition is wrong.

If it energizes you more than the short time it took to do it, then you should do as much of it as you can!

Asking ourselves, "is this thing I'm considering doing the best possible thing?"

But that's an impossibly high threshold to clear, and if you try, you'll likely get frozen in place, unable to act.

The real threshold to clear is not "is this perfect" but "is it better or worse than the actual alternative if I didn't"?

The actual alternative is almost certainly "the status quo."

This bar is significantly easier to clear.

If you follow this, you might have some arcing / circuitous paths and wasted effort, but you will be able to be moving, adapting and learning.

Similarly, science as an enterprise isn't about finding absolutely true, black and white facts.

It's instead a social enterprise to continuously find more-plausible theories to explain the actual evidence.

An ever-more plausible collective explanation of the world.

"I'm being data driven, using the output of this objective process."

"Yeah, but you're cherry-picking the 1% of signal out of the process's output that confirms your priors..."

It's OK to use your judgment to select examples, but you'll tend to pick things that already support your priors.

When there's a lot of diverse output to choose from objective processes, there's a lot of opportunity to find disconfirming evidence and learn.

... But there's also a much broader field to cherry pick from, making this cherry picking effect even stronger.

It was simply a thing that must be so.

What's the chance that Apple won't be around in a decade? Very slim.

What's the chance that Apple will be around but Apple Wallet won't? Very slim.

What is the chance that ID cards won't make sense in Apple Wallet? Very slim.

Armed with that clarity, you can embark on an even decade-long slog with conviction.

This is something that aggregators can do easily, given the gravity well dynamics of their business.

It's not that someone at Apple was lazy to not do the TAM analysis.

It's that the TAM analysis could not possibly matter given the fundamental characteristics of the domain.

The gravity well dynamics obviates all of the other analysis you might do.

This is good: it's what allows them to adapt and create.

But it's also bad: there's nothing stable that others can use as a foundation to build coherent things on top of.

Building is about creating non-alive structure as a foundation.

This structure stays put in place, a seed crystal of permanence for living things to anchor (temporarily) on and build something coherent and stable.

Structure and living things are both necessary.

Structure however can box in living things, when it prevents the living thing from spreading out, or growing upward.

This is a box, a cage.

Structure that is a platform or trellis allows living things to grow upwards further than they could without it.

A platform that is elevated to start from, or a trellis: a ladder to grab onto and pull themselves upwards.

Good structure supports; bad structure constrains.

The best structure is a living structure, similar to the vine bridges in India that are woven over generations.

Platforms support from the bottom, lifting up possibilities.

Aggregators support from the top (cornered consumer demand), lifting up possibilities.

But aggregators fundamentally set a ceiling: "As a participant in my ecosystem you can do anything up to this level, but no higher."

A true platform doesn't set a ceiling of possibility on its ecosystem.

An aggregator, like a platform, creates the opportunity for an adjacent ecosystem.

But unlike a platform, an aggregator is closed.

As is often the case, Gordon said this better two years ago.

Which requires maneuvering room to make decisions.

Without exercise, your ability atrophies.

Protocols here could be APIs, or other formal agreements, like laws and contracts.

If you have only the explicit protocol but not the necessary tacit knowledge, you might not be able to use it effectively.

Sometimes the necessary tacit knowledge is minor. Sometimes it is major.

The tacit knowledge is often passed down culturally, or through apprenticeship.

Even seemingly structured protocols can require tacit knowledge.

E.g. SMTP has been around for decades, but to actually effectively deploy it requires you to know all of the little idioms and oddities of real use.

Sometimes you can make hidden tacit knowledge more explicit.

For example, Ian Hickson cleaned up the arcane / under-specified HTML parsing specs a decade ago.

He did it not by actually cleaning up the semantics, but rather documenting, in excruciating, concrete detail, how things actually worked.

This converted previously tacit knowledge into a formal protocol.

However, often the tacit knowledge cannot be made explicit, or the whole thing shatters.

This can happen for example with convenient social fictions that lubricate day-to-day social interactions.

You have to abduct your intuition to turn it into language.

The dumber the student, the more it requires you to abduct your understanding.

Nothing is dumber than a computer.

So explaining something to a computer so it can execute it is a great forcing function for you to understand the thing well enough to do a predictive model of it.

(This line of reasoning originates with Peter Norvig, I believe.)

Too high and you preclude lots of possibilities.

Too low, and you leave a ton of things to be decided as convention in userland.

That can be great, but then if multiple things in userland need to coordinate they can take less for granted and figuring out how to coordinate might be hard.

In general it's a good idea to have at least a conceptual understanding of the bedrock layers as platonic ideals.

Often primitive archeology is hard because you're trying to post-hoc rationalize a constructed/built object out of an organic, fractal, wrinkled thing... an exercise that is not guaranteed to be possible.

You don't need to expose those semantics to start, but if you have a consistent concept of them that you attempt to maintain wherever possible as you evolve, then you retain the option value to dig down and do easy primitive archeology because you know that there are coherent primitives to be unearthed.

An idea that pencils is plausibly viable. That means it might work.

If you can't get a big idea to even pencil, then it definitely doesn't work.

The real world is much less forgiving than the drawing table.

This is only true for ideas that are already big, where the question is the downside of how likely it is that it doesn't work.

For ideas that start small but might grow into something much bigger, the value you're able to figure out on the drawing board will undercount the upside.

Yes, the frame sounding like toilet paper is intentional.

You want multi-ply thinking; thinking that is at least second-order.

If you think a ply deeper than your adversary (a step further), you can beat them.

One-ply thinking looks rigorous but is actually dangerous and misleading.

It would be like strategizing on a chess board but assuming that all of the other pieces will stay fixed in place.

If everyone around you is thinking one-ply and you're thinking 3-ply, then you have an edge, but you're still not thinking strategically.

Just being taller than everyone around you does not make you tall.

This is another smuggled infinity: "[We are infallible, because we're the main characters.] We decided X, therefore X is right."

But if X turns out to not work in practice, then be open to the idea that maybe X wasn't right?

Maybe you're actually fallible, like every other human?

E.g. in The Emperor's New Clothes, the system is in a supercritical state.

Everyone can see with their own eyes that the emperor is naked, but no one wants to point it out lest they lose their head.

But all it takes is one child laughing to set off a cascade of mutual awareness.

Before a supercritical system shatters, it looks impossible that it will ever explode.

After a supercritical system shatters, it looks inevitable that it would explode.

Pond scum has a kind of agency. But in a way that is foreign and distributed.

Where is the center? Where is the brain? Where is the little person making decisions that I can talk to?

We anthropomorphize these systems.

But they are actually more resilient if they don't require a central point.

The whole can survive even if any individual part of it dies.

These are swarm intelligences.

Totally alien, but not because they aren't common, but because we can't talk to them, so we don't think about them as a collective entity.

LLMs are like a pond scum; it feels like we can talk to them, but this is only a tiny fraction of what they can do, and it limits our imagination of how we might use them.

Google is extremely impressive at figuring out how to interpret your query.

It almost feels like it intelligently understands your query.

But it doesn't! It's a carefully crafted illusion that cleverly stitches together the piecemeal insights from a very large group of users into the wisdom of the crowd.

Google is a mirror that focuses the wisdom of the crowds but is not itself wise.

To use it well, you have to formulate a query that is likely to overlap with content and queries other people have done before.

"How would other people structure this question?" to ensure you can draft off what happened before.

Contrast that with LLMs; they have absorbed a kind of reasoning about the content; a wisdom of the crowds, but also its own kind of emergent wisdom.

That means that you can talk to LLMs more like you talk to a real person.

This happens with for example sandboxes, which preclude dangerous uses.

Without a sandbox, there's no limit to what you or a developer can do... but there are a lot of ways to blow your foot off!

A sandbox makes it so the downside risk is significantly curtailed.

This makes it significantly safer (and thus cheaper) for people to explore and experiment and find new pockets of value that otherwise wouldn't have been discovered.

The web did this with a deliberately curtailed security model.

Critically, the web did this with a bare minimum of centralization--quite a feat!

This means that the web is a proper platform with no ceiling.

Apple has done something similar with the App Store.

But Apple has done it with extreme centralization.

Apple's an aggregator, with a strong (and low) ceiling set.

In the world Apple has created, Apple is god.

And as they have shown in their bad-faith response to the DMA, they are a vengeful, jealous god.

He is not a chemistry expert, but is now able to consume relevant chemistry papers that previously would have required a PhD.

He told me he "literally could not do this job without ChatGPT."

ChatGPT makes it 10-100x easier to slog through papers from a field you don't know.

Imagine having a friendly graduate-level instructor on literally any topic in the world come sit next to you to help you interpret any paper you're reading.

He thought he was the one; the person with infinite agency who will fix the system from outside of it.

But later he discovers that he is actually just the most recent "one," and that the system is designed to regularly produc "ones" as a pressure release valve.

That is, he is not free of the system, he is a part of it.

But once he embraced that frustrating truth, he could lean into it.

"How am I different from the ones that have come before me?"

And then once you figure that out, he could lean into that difference, and ultimately change the whole system from within.

Navigating complexity is easy. It simply requires letting go of the comforting illusion of full agency that your ego doesn't want you to let go.

Imagine collaborating with someone who you dislike. What do you do?

Imagine if you're living in a small town two hundred years ago.

You will almost certainly see them again repeatedly. You'll have a hard time escaping them.

So it makes sense to learn to live with them.

You'll likely invest in understanding their position, finding a dynamic tension to live with them in.

As a result, you both grow more flexible and learn how to live together.

You both get stronger, and the community does too. You both grow from the challenge.

Now imagine it's a random person on the internet on a pseudonymous message board.

You'll almost certainly never see this person again.

It's easier to just discard the relationship like a tissue and move on to something less challenging.

Both of you do not grow.

You're left the same as before, and possibly worse.

Instead of thinking "I don't like Bob's point but I see where he's coming from", you think "Bob was a jerk" and that's that.

What will happen to society as more and more of our relationships become tissue-like?

Imagine a prompt like "Write a synopsis of X, and bold the most salient words."

The LLM has to choose to emit the markdown bold characters and then output the word.

But the LLM is effectively two separate invocations, two separate entities for generating that asterisk and then for generating the important world.

On the second iteration, how does it know what the first iteration was thinking?

How can it act as a single coherent thing when it's actually multiple things?

Imagine 5 friends sitting down and being told to generate one picture in five slices.

The first friend will draw the first vertical slice, then next one the next, and so on.

They won't be able to talk or communicate.

The first friend draws a bunch of guns pointing to the right.

The next friend sees that and goes "I guess there's something scary here... I'll draw a dragon."

The next friend sees those first two slices and goes, "OK, there's a scary monster here, maybe it's because the dragon is guarding a cave of gold."

Each friend is operating entirely independently and trying to add something that is coherent with what came before, without any explicit coordination.

But the result, if everyone collaborates, could look 100% coherent.

The key thing is that as long as the entities doing the components are fully operating in good faith and earnestly trying to add a thing that is coherent with what has come before (and what will likely come after--if you're doing a collaborative poem and someone ends their line with 'orange' they're just being a jerk), you can get the illusion of a pre-planned result, when it really wasn't at all.

Another mental model: LLMs are like the main character in memento.

The main character "wakes up", sees the tattoos and the sticky notes, and says "well I guess the next step is I'm supposed to call Teddy."

He does that action and then blanks and then wakes up and says "the phone is ringing, I guess it's Teddy i"m calling".

The LLM is in some ways little slices of independent agency, but they're coordinating by leaving tokens in the shared working space (stigmergy!) that they assume are to be trusted as inputs.

By the way, this is not too dissimilar from how our brains actually work.

(Not in terms of 'each token is a totally new computation from a blank slate' but in terms of 'lots of independent neurons cooperating to give the illusion of coherence')

The different parts of our brain are constantly working to maintain this consistent illusion of a coherent self.

E.g. the left brain post-hoc rationalizing what the right brain decided to do in split-brain patients.

Ian Couzin's research in how fish schools decide which target to navigate to shows it is precisely the same algorithm that our neurons use to coordinate as a group to emergently make the same kinds of decisions.

We feel like a coherent entity, but we're closer to a slime mold than we think!

This is not too dissimilar from how plain-old-code works, either.

The code is executed, and looks in its protected storage partition and thinks "Well I guess this user is named Alex Komoroske, because there's a note here that says that, and I'll presume that no one else has access to this locked storage room than previous iterations of me."

A corollary: the "identity" of a computing experience is code+data (data here including anything still resident in memory from the last invocation).

If you change the code, or the data, then it's a different "thing", operating with different agency.

A key observation: a centralized LLM in the cloud that has memory about you is potentially very scary / powerful (it could keep track of information about you to subtly manipulate you in the future) .

But the very same LLM in the cloud, that is known to not log any state, is significantly less scary, because each token you're effectively dealing with a whole new entity.