Bits and Bobs 1/13/25

It is up to the human co-creating with them to drive them to something interesting and curated and good.

LLMs have no filter for what is good.

You need humans in the loop to apply taste and select the good stuff.

The quality of output from an LLM is highly correlated with the quality of the human using them.

LLMs give great guacamole recipes, because they've seen every guacamole recipe and can triangulate what they all have in common.

They give you higher leverage.

Will that cause you to be satisfied with doing less, or driven to do more?

The mindset and concepts you bring to the LLM determine whether it gets good results.

If you have a coherent worldview and concept, the LLM can help you fill it in.

If you don't, the LLM will hallucinate the concept for you.

The editor's mindset is more important than the writer's mindset in this process.^[ajr]

Someone was telling me they've started writing poetry again after 20 years, because it's so much easier to co-create with an LLM than to do it alone.

We'll see an explosion of amateur poets and musicians co-creating with AI.

Even if no one reads their poetry, the act of writing poetry improves the world.

An amateur poet becomes more observant of their world, they become a better, more engaged, more thoughtful participant in their environment.

The human and the computer, the distilled hive mind of all of society, in a co-creative process.

Fundamentally different, fundamentally better, than a computer talking to a computer, or a human talking to a human.

The tension and difference in skillset is what produces novel insights that could not have been created with a pair of similar conversants.

Intellectual GPS.^[ajs]

When you have a co-creative muse, you never feel lonely or lost in your creativity.

Any question you ask it. it will reward you with some thought.

Kids should get used to "have some curiosity? Go ask the LLM!"

A massive complement to "sit here in your seat and do rote memorization to pass the test."^[ajt][aju]

Embracing the joy of discovery.

Curious people (especially ones with a need for cognition^[ajv][ajw]) will keep pulling on threads and will love them.

When Google was a new thing, people could have "Google-Fu" who had developed a calibrated intuition for how to extract better answers from it.

But as Google matured, a clear gradient of improvement is to make it so less-and-less savvy users get results closer and closer to what users with Google-fu can get.

The main way to do that is to automate the intuition^[ajx][ajy] that the savvy users are applying; which now means everyone gets it, and the edge evaporates.

Prompt-fu is the equivalent for getting much better results from LLMs.

That implies that people with prompt-fu will get less and less differentiated results as the systems we all use get more mature.

But presumably there will always be some difference, because a search is just a filtering operation, whereas LLMs are co-creative, so the "Fu" is orders of magnitude more important than it was for search queries.^[ajz]

Apparently if you train them on RNA sequences to images of the rendered protein, they do a surprisingly great job at predicting what a given sequence will fold to.

LLMs are tapping into a hidden structure of the universe that reveals itself only if you are patient enough to sift through it.^[akb][akc]

LLMs are patient, and observant enough, to tap into that structure.

For example, how much does it matter to use a model that implicitly knows an elephant is bigger than a mouse?^[akd]

You can think of summarization as a process to factor out the background knowledge a reasonable listener would take for granted, leaving only the "diff" of interesting meaning.

That implies that the more the LLM understands about the world what a typical human does, the better the summarization will be.

Presumably that ability has a clear logarithmic shape, where more background knowledge gets less and less useful.

That information helps the LLM perform better, but how much?

Someone pointed out that the Shapely Value might be a useful conceptual lens to try to get a handle on this.

That is, lots of options, good price, and quality competition.

That's an amazingly optimistic result for society!

We can take GPT-4 level quality for granted.

If you get just a single english-language append-only log designed for a human, you make the LLM take its rich understanding and distill it out through a teeny straw of a single, human-understandable line of thought.

Techniques like test-time compute in O1 and similar models allow the model to spray out lots of low quality ideas and then refine.

The quality of an LLM's juice is tied to who is doing the squeezing.

An implication of all outputs from LLMs being co-created by the human and the LLM is that different humans can get wildly different results.^{[akg][akh][aki]}

But a lot of discourse about LLMs doesn't differentiate the two.

There's a difference between an LLM in training absorbing a hologram of the knowledge vs RAG to help sift through concrete input with its background common sense it absorbed in training.

Sometimes you just need its background worldly knowledge to give it common sense.

If you want details, that's not sufficient and you''l need RAG.

Adding more knowledge to a model is expensive, has long lead times, works on vibes and is imprecise.

RAG can't give huge context to a model that doesn't have the right background knowledge, but it can be updated quickly and can enable precision in details.

Everyone talks about these things like they're the same, but they're wildly different.

Training your own model is very capital intensive.

But in many cases you can use an off-the-shelf LLM plus RAG and produce amazing results.

The question is: how much background knowledge do you need for the LLM to have enough common sense to be able to tackle your concrete tasks where you bring the specific details for it to operate on.

The centralization is implied due to the capital requirements of training and inference, the existence of proprietary models, and the efficiency of scale in serving.

Centralization would be bad if there were stickiness to models.

But if you treat the model as a dumb, stateless machine, giving an answer to your prompt and then forgetting, it doesn't matter much; they don't accumulate data to accumulate power.

It's once the model starts getting a memory^[akk] that the power dynamics turn into something possibly compounding.^{[akl][akm][akn]}

The fact that LLM providers are now commodity and also that the API is the same and easy to swap to a different provider at the flip of a switch helps reduce the likelihood of centralization of power.

Capital intensive, but commodity.

But actually LLMs are a worse business than electricity providers^[ako].

Electricity providers typically have a geographic monopoly; end consumers only have one option to pick from, which gives some pricing power.

But LLMs providers don't have that, it's super easy for someone to swap providers in a second.

It's even the same basic API (prompt -> response).

It will cause an app unbundling^[akp].

It reduces the "transaction cost" of creating software, which, similar to the Coasian theory of the firm, sets the efficient "size" of bundles of software.

You won't see the apps^[akq] much... they'll still be there, just kind of boring basements below the layer that you the user spends time in.

But that implies that you have agents who can see and intermix data across apps, and that you trust to not take incorrect actions on your behalf.

That's a high bar to meet if there's even a little bit of downside risk... and if they're flexible and open-ended there's always downside risk!

Another approach is not an over-the-top of existing software, but new software emerging.

Perfectly bespoke software on demand.

Sprinkle pixie dust on your data, it comes alive^[akr].

The software is the least important part.

The software is a means to an end, an implementation detail.

Something that emerges, that you can take for granted.

Instead of making the data look pretty, it makes it animated, interactive, useful.

Instagram filters give you just the right amount of agency.

You feel like you are making good decisions... but the decision space has been constructed so all of the decisions you could make are good.

You are imbuing it with your taste with high-leverage tools.

Even if you don't realize what you're doing, and would never consider it "programming".

Curation and micro-decisions as an act of creation.

Supercharged with AI, it allows software to emerge, implicitly called into being just in time.

One with different rules, a different gravity.

Where things that in today's world would be considered magic are ordinary.

If you want to grow your audience, you have to get closer to the lowest common denominator.

This is inescapable.

An alternate way to express Ivan's Tyranny of the Marginal User.

The way that traditional software is made is fundamentally and intrinsically bad for you.

Organic software should work just for you, and should be healthy.

Email is extremely noisy, but also tons of signal.

What if interactive, useful software just for you could emerge out of that compost heap?

At places like Google it's impossible to do this because although they have your data^[akt], they would have to build, not grow, software.

The true "yes, and" is "no matter what you say, I'm going to embrace it and make you look good".

"Yes, and" should be building on top of, co-creating with the other^[aku], a thing that is better than what either of you could do alone.

But others get to decide if they want to play in your game or not.

This naturally balances out incentives.

You have to do something that others will want to participate in.

Like the optimal process for fairly splitting a dessert that all kids know intuitively: one kid splits, the other kid picks.

The seed crystal exists because the first few people who looked at it found it useful.

The boundary gradient works because people at the margin can see that the thing is useful and is not hollow.

The quality of the thing thus does correlate with how successful it becomes, but the quality to outcome is not linearly proportional (due to the compounding being a self-accelerating phenomena) but a power law.

That's why power laws happen; the boundary gradient's steepness is driven by quality (a linear difference) but the size of growth is driven by surface area of the boundary, which grows with the square of the current size.

Benchmarks are an emergent schelling point.

Someone sets rules and a way to measure quality.

No one has to use their rules if they don't find them valuable, but if people do, then other people will also want to show they can do well on it, which is a compounding loop.

People take it seriously because other people take it seriously, and people take it seriously because every marginal person who considers taking it seriously looks at it and agrees that it sounds plausibly useful enough to take seriously.

Benchmarks can get a compounding amount of momentum in proportion to their quality.

Because data is non rivalrous, it doesn't suffer from the tragedy of the commons.

Only atoms have the tragedy of the commons, not bits.^{[akw][akx][aky][akz][ala]}

Bits are non-rivalrous in their consumption; you can make infinite perfect copies for free.

So internet bandwidth can have the tragedy of commons, but not the information carried over the wires.

Digital commons tend to get more investment from an ecosystem the more use they get.

The more that others use them, the more that people are invested in making sure information they care about (e.g. information on their hometown) is high quality.

People invest in proportion to how useful other people find the resource to be and how much traffic it gets.

So the more people use it, the more value is naturally created.

You could argue that because there are benchmarks for e.g. LLM safety, LLM providers will also compete on doing well on those, naturally.

Safety is a thing that people want to want, but don't actually want.

If people get an incremental personal benefit from using an unsafe system vs the competitor who uses the safe system, they'll take it.

That leads to an arms race on capability leaving behind "safety."

"Well if I, the good guy, don't push the limit, the bad guy definitely will and then dominate us all, so I need to push the limit so the good guys win."

The co-evolving set of competitors hold each other in check.

Competition sets co-evolving constraints that are balanced and just right to encourage a gradient of improvement that pushes all competitors to excel.

But it's a bowl on a pedestal kind of dynamic; within the middle range the competition is self-balancing, but if you get out of the place where it's possible for competitors to catch up, the engine that leads to quality growth leads instead to run-away power accretion.

By using the power-accretion gradient and aligning with quality improvement, everyone improves due to individuals' greed.

But greed without the constraints from effective competition consumes everything.

It is only the competition, the dynamic equilibrium, that drives to quality creation, the positive-sumness of greed.

Without competition, greed just consumes all and puts everything into a fully captured heat death kind of static equilibrium.

It's better able to invest in things that are working; but it can get much harder to iterate and experiment and see what's working.

Financialization can make it very hard to get the seedlings to be able to garden in the first place.

Financialization is about hyper legibility, hyper focus on quantitative.

But sometimes you don't need it, because it's more expensive to create legibility than it is to just do the thing.

In the time it would take to document a seedling, you can plant 10 seedlings.

As long as the downside risk of the seedling is capped and small, it's just opportunity cost, and making it fully legible just kills the seedling before it even gets started.

The most of just whatever our lizard brains want.

Being chased by a rabbit. A kind of weird yell at a rally. Misspelling potato.

Why are they so much more tame?

I think it's because of the internet.

The information you are exposed to daily sets your baseline for normal, for what is exceptional or stands out from that baseline.^[alb]

Before the internet, most information went through curatorial processes optimizing to make them normal / balanced.^[alc]

The internet makes it so there's infinite channels for information to flow, and also a constant drive to produce things that are "more".

The things that break through the cacophony are memeable, the "most".

That means we're awash in a cacophony of over-the-top, supernormal stimuli and that becomes our new normal.

Now that the baseline is over-the-top, it's the shameless who have adapted to the new reality the fastest.^[ald]

The shameless have realized that even egregious behavior barely registers^[ale], so the downside is much smaller, and it's a much better strategy than before.

Before, doing the right, principled thing, and the thing that was most effective in repeated games were aligned.

Now, they are less aligned, and the shameless reign supreme.

I truly hope we as society figure a mechanism to realign principles even in this new information reality.

The rules don't follow themselves or apply themselves.

They require people who believe in the rules to hold themselves to them or apply them to others.

There's no regulator to slow down the process, especially if there's a moral fervor about righting some persistent wrong.

Everyone is swept up in the momentum, even as they increasingly lose conviction that it's still justified to the extent the movement is executing on it.

Going against the momentum, against the stampede, now becomes dangerous (you'll get trampled) so you go along.

Only after the mob has clearly and ambiguously crossed the line can everyone go, "oh, yeah… that was obviously too far."

Unfortunately that tends to invite radical over-reactions from the other side.

The thing that is most prominent, most distinctive, wins by default in the chaos.

I was talking to a person at CES who had an amazingly lucky break: a job offer for a cool and extremely differentiated role.

She got it when she was the only American living in a particular region of China.

She stood out, which meant she was structurally more likely to get noticed.

When you have a consistent, asymmetric edge in a dataset and average it across many, many, many iterations, the edge pops out, clear as day.

The more data, the more convergence and revealing the true, consistent edge, no matter how much noise.

It's not the end thing, it's the catalyst to propel you to the main thing.

It's combustible, you couldn't get it started without it.

But it is consumed in the process, to create the fire to get the actual logs going.

The prototype is ethereal, self-immolating.

It's just about consuming itself to create momentum.

Don't focus on making it perfect; focus on making something combustible.

People wanting to use it is the most important one.

You can't force people to want to use it; it has to emerge, organically, if you've built something that is viable and useful.

Lock the parts down you can, getting a larger and larger base of concreteness.

That foundation will allow you to reach further into ambiguity.

You need something concrete to lever off of to stay strong in ambiguity.

A mistake a lot of people make is to focus on the ambiguity instead of what can be easily nailed down.

Ambiguity has compounding cost, so nailing down more pieces makes remaining decisions orders of magnitude cheaper.

Nail down the most obviously true, inescapable parts first, and work your way down to ever-more debatable points.

As you work your way down and get more real world feedback, it will get easier and easier per unit ambiguity.

Don't postpone decisions you don't need to.

In ambiguity, when you feel weak, you get more tight, defensive.

The defensiveness creates brittleness and makes you less resilient.

You need to feel strong to act strong.

A kind of inherent chicken and egg that can lead to a vicious or virtuous cycle depending on your natural disposition: how optimistic you are.

You can't say "I don't know."

So you need to go to increasingly ridiculous, unproductive lengths to avoid saying "I don't know.".

Even worse than needing to have the answers is needing to be the one who comes up with the answers.

Being able to embrace that you don't have all of the answers is a strong stance that requires self-confidence.

Acknowledging "I don't got this" takes a lot of self confidence and trust.

This is one of the reasons the two unteachable skills need an optimistic, forward momentum: to get that chicken or egg self-confidence and strength coevolving earlier.

When it's easy, when it's not a chore to push beyond your current ability, to apply discretionary effort, to go beyond.

When it's effortless to excel is when greatness can emerge.

When you don't have the muse don't worry about it or try to get it to show up.

When it strikes, push it as hard as you can and be thankful for it.

Don't stress when it's gone, it will come back, and you can't force it to come back.

All you can do is make the most of it when it does hit.^{[alh][ali][alj]}

Simple is how hard it is to understand the right answer.

Easy is how hard it is to execute the right answer.

It's totally possible for something to be simple but not easy: a slog, or an unforgiving domain where the action is easy but most attempts end in failure and the only way to get good is to develop the knowhow and feel.

Then, unexpectedly, magic happens.

In a flash you realize: this person was not flailing, but doing intentional, precisely controlled steps the whole time.

A perfectly executed magic trick.

Every seemingly random movement was dialed in to call the magical result into being.

You have an 'aha' moment, as the bisociation collapses.

You go from thinking "this person is just making it up as they go along, with a lot of noise" to "this person is in very tight control, and I should pay attention to every detail^{[all][alm][aln][alo]}."

That aha moment can be very compelling and make you trust someone deeply.

When kids are learning to talk and they're just talking to their parents their parents understand them even when they're unintelligible to others because the parents remember what that sound signifies from direct prior experience.

The parents are reacting less to what the utterance sounds like and more recognizing it from what they know it's connected to.

If you say a thing and it's unintelligible to the receiver, you have to modulate and try again.

If the receiver understands it, there's no need to modulate, because it achieved its goal.

But being with a bunch of peers, if they can't understand you you need to change how you speak.

It's only by failing to be understood that the child has an incentive to produce the utterance more clearly.

English is an equilibrium of being mutually intelligible.

So kids at school have much better language production.

They have to be intelligible to a lot of other people who don't have the time or mental capacity to remember every random person's specific speaking style.

The slack helps absorb surprise without a compounding cascade of thrash.

It also allows little seedlings of greatness to grow.

Don't think of slack in a system as "employees sitting around twiddling their thumbs" or "doing unimportant hobby projects" or "playing foosball".

Think of it instead as "doing preemptible work that is important but not urgent."

Doing the P2s. Which, over time, can add up significantly.

In a small company, you can be efficient but you can't–and shouldn't–be overly thorough.

In a large company, you can't be efficient, and you should be thorough.

The worst of both worlds: inefficient and not thorough!

That can easily become navel-gazing and apportioning blame.

Focus on how to get the most out of where we are, given that we're here.

At 1000+ it's 5 days a week.

At 100 it should be 1 day a week.

Up to 40 people it should be zero.

If you're under 40 and it's more than zero, something is wrong.

Politics emerges in every assemblage of people, but it should not become a dominant force until you've grown to a size where not everyone can have a direct, personal trusting relationship with each other person.

Even if the people are very different, they can have fun together.

People who are defensive or trying to be something they aren't, it's not fun, it's like swimming upstream.^[alp]

Not a cage, a scaffold.

Some environments already have some structure you can take for granted, and all you need to do is add fertilizer.

But in early stage, pre-coherence contexts, you can't take any structure for granted.

In the world of bits, you can go from a rough idea to a rough prototype often in just a few hours.

This makes it much easier to explore, find compelling ideas, and experiment.

In the world of atoms (e.g. hardware), it can take many, many, many orders of magnitude longer between having the idea to determining whether it's actually promising.

The longer between the decision to invest and determining whether it worked, the more like roulette it is.

Survive, then thrive.

When you're in your highest and best use, you're in your flow state.

Sometimes the context is a thing you can control, but often it isn't; you need the support of others to help make that context correctly aligned.

Sometimes the context that two high-ability people need to thrive are simply incompatible.

"I don't like it!"

"abrasion becomes tension if the other person isn't ready for it

nobody wants a scraped knee / ego, so more often than not it ends up being tension anyway

trust is the antidote"

Point at it, hold it.

Acknowledging it allows you to hold it at arm's length and move it to a has-a relation.

If you don't, then it is implicitly part of you (is-a) and can have strong implications for how you act that are often not what you want to want.

Don't ignore emotions. Name them, label them, hold them as an object.

Failure is the only way to learn.^[alr][als]

Failure hurts.

Don't try to make no mistakes.

Try to fail at the fastest positive rate to learn.

Fail early and often.

Smaller mistakes that are less likely to knock you out of the game, a faster feedback loop to learn.