Bits and Bobs 7/29/24

For example, one of the big benefits of a wedding planner is not that you offload judgment decisions to them, but that you know they'll come back with a handful of reasonable (in terms of quality and cost) options to pick from.

This kind of assistant is very forgiving; they don't take actions on your behalf.

The human judgment is in the loop, always.

But the boring / hard part is automated away.

Google Search is fundamentally framed as "here are 10 options, you pick the one you want"

Very forgiving UX modality, with a clean hill to climb of quality.

4-up UI is great for coevolving an answer with an imperfect assistant. Dancing with the LLM.

But once the open system catches up, it never falls behind again.

The only reason to use the proprietary option is quality; all else equal everyone else in the ecosystem would rather go with the open option.

Less lock in

More competition on price and reliability

The option that everyone else will use too, a natural schelling point.

You want to use the option everyone else will use because it's the one most likely to get investment to improve.

This is tremendously exciting!

A few months ago we had only one existence proof of GPT4 level quality.

It was totally possible that we'd only ever get GPT4, which would lead the ecosystem to a very different, highly centralized outcome.

A month or so ago we got Claude Sonnet 3.5, and now we have an open weights model in the same ballpark.

The open system has caught up!

If you had to bet on a single model family today, Llama is now the clear bet.

Interestingly no one has exceeded GPT4 level quality.

Will we ever exceed it, or is this a natural ceiling?

All of the labs are very confident we'll exceed it.

But then again, it's their directly vested interest to believe that there is significant headroom in quality… and to get everyone else to believe that, too.

If it's a natural ceiling in quality, then from here on out it will be all about efficiency.

GPT4o-Mini is already spectacularly cheap.

I kind of hope that we do hit a ceiling somewhere around GPT4 levels of quality.

No risk of runaway AGI.

But tons and tons of capability overhang for society to harvest and figure out how to use for the next few decades.

E.g. GPT4o-mini is so cheap and fast, you can run a number of different iterations on a prompt in parallel and then pick the one you like the best.

Or let the user pick if you have a 4-up evolution style UI.

Full automation requires a human to not be in the loop.

Perfect reliability requires a closed loop system that can learn.

LLMs today are an open loop.

The queries might be saved and used in some fuzzy way to train future iterations of the model, but they don't affect the LLM as it exists.

The more general the tasks that users are expected to hand off to the system, the harder it is to be perfectly reliable.

For users to trust it the quality has to be resiliently / broadly good enough.

If you didn't make a mistake, there's nothing to update in the model.

You've made a lot of decisions in the past to calibrate from, so you develop a pretty good intuition for your preferences.

But an assistant (a person, or a computer) only has limited examples to draw from and calibrate from.

This is one of the reasons that "partial executive assistant" services and computer based assistants are often underwhelming, and hard to hand off meaningful work to.

The harder it is to find those truly great needles in the haystack, the more incentive users will have to share them (and get social credit) when they find them.

Imagine a system where good-enough items are relatively easy to find, but truly great ones are hard to find.

The ease of finding good enough items keeps lots of people searching for needles.

The challenge of finding great ones makes people more likely to want to shout from the rooftops when they find one.

When people see other great items people are finding, it makes them more excited to search for great items, too.

This creates a viral loop in the ecosystem.

Then if you can remember that choice for other users, too, then you get a system that continues to improve.

A self-improving system with the LLM and humans working in concert, better than either alone.

The LLM is great at coming up with mostly reasonable options quickly.

The human is great at applying judgment.

For example, imagine asking Alexa to buy you something.

You're basically saying, "do a search for that topic, click the top result, and buy one."

This implies that you really trust the quality of that search result to give good results and have the best option as the first position.

How willing you are to take this leap of fate is tied to:

1) How good do you expect the search results for this query to be

Based on how good the search results in general have been in the past

As well as how good you expect the service to do on this query in particular.

2) How big the downside for a miss is

If you're buying a $100 item that will be shipped to you, the downside is quite large; you're out $100 and only realize it was wrong days later.

If it's a Google Search, the downside is tiny; just immediately do the query again manually.

Amazon's search results have famously become quite scammy and crappy, which means trusting Alexa to buy an item for you has gotten harder.

The common AI assistant use case of "plan a trip for me and automatically buy tickets" is almost impossible to imagine working in practice.

Both because it's hard to imagine high enough quality

The preferences we have for travel are subtle and hard to explain to a human we know well, let alone a machine.

And because the downside cost is so high.

"First, to ensure that we have access to the best technology and aren't locked into a closed ecosystem over the long term, Llama needs to develop into a full ecosystem of tools, efficiency improvements, silicon optimizations, and other integrations. If we were the only company using Llama, this ecosystem wouldn't develop and we'd fare no better than the closed variants of Unix."

Compounding loops take time to "warm up".

For a critical period at the beginning, the linear system will beat the compounding system.

Proprietary internal investment from one entity is linear.

Open external investment in a swarm is compounding.

Open systems start slow but win in the end (perhaps a very long time down the road)

I like how Gordon distills it:

"Innovation is combinatorial. Each unrelated discovery expands our possibility space by increasing the inventory of components we can combine to create something new."

It all comes down to: which system can try more combinations more quickly?

A system that sums the investment of all participants will beat a system invested in by only a single participant.

"The word I came up with to describe the (strange, unintuitive) fact that state of the art LLMs can both perform extremely impressive tasks (e.g. solve complex math problems) while simultaneously struggle with some very dumb problems."

As with the "Which is bigger: 9.9 or 9.11" question from a few days ago that led Andrej to his "jagged intelligence" tweet.

LLMs are like system 1: a general approach that is surprisingly good at vibes-based processing for everything, but not particularly great at anything.

But there are lots of sub problems that can be resiliently correct, a kind of niche system 2.

For example, plain old computers are extremely good at arithmetic.

Why try to brute force everything with a system 1 type approach?

Why not plug in adjacent systems for some subdomains like arithmetic, with a function-calling style interface?

So a good formal grammar constrains the generation and allows generating lots and lots of good examples.

The new math AI paper, shows that the more formalizable the system and rules, the higher quality the training, the better the inferences.

The problem is that you get energy not when you need it but when there's supply.

But there are some kinds of workload that can be offset in time to when there's cheap supply, e.g. LLM training or BitCoin mining.

BitCoin mining has no incentive to get more efficient; the proof of work (which scales with electricity) is the whole point that undergirds the trust in the system.

AI model training, however, everyone's incentives are aligned to reduce the cost of the energy use.

Preferential attachment is a kind of emergent law of the universe, like the inverse of entropy, a thing that shows up inescapably.

When evolving an ecosystem, if you don't design any points for beneficial aggregation, you'll get parasitic aggregation outside your system, in a way you can't control or influence.

But LLMs are great at duct taping together not-too-complex software on demand.

Everyone (assuming they are fully informed and have time to consider the decision) would prefer the more private option, all else equal.

That's not to say this edge is very large; if the more private option is even a little more inconvenient, it might not be a viable choice for users.

If some options add intrusive logging, but they are easy to replicate and small, then there will be a competition for ones that have less logging.

It will be easier to create a drop-in replacement for the privacy-invasive alternative, and once the same functionality exists without tracking, users will over time shift to it.

That is, to see the data in its full fidelity, with the ability to summarize, store, log, transmit?

If you as a user copy/pasted a complex Excel formula from StackOverflow, should the function author get to see your data?

Obviously not!

If you installed an app, should the app get to see your data in the app?

Of course!

The app is the controller of the data.

Apps must convince the user to trust them with their data before being installed.

But needing to trust the app is also a limiter of apps being installed in the first place.

To be installed an app has to be useful enough for the user to take the leap of faith; to be useful enough it has to glom together enough use cases (plus a viable business model!) to convince a user to install it.

This leads to chunky apps: apps that are larger than they could be, and monolithic, one size fits none.

What if you could make it so authors of code didn't get to see the data?

They wouldn't need to earn the trust of the user to run, because they can't do anything with the data like transmit it back.

If a tree falls in the woods and there's no one there to hear it, does it matter if it makes a sound?

This would allow much smaller bits of code to be viable.

Much cheaper to produce.

In fact, they might be so cheap that only the energy of hobbyists and tinkerers is necessary.

Scarcity is relative!

If you decrease scarcity of one part of the funnel, another part becomes the new bottleneck.

Software has been scarce because it is costly to create.

What happens when software is abundant?

Especially if there's an undo button or the stakes are low, so the downside for a failed experiment is low.

Expose too little, and your platform won't get used and might get left behind.

Expose too much, and you'll have significantly constrained your future agility.

Every time you want to change the API, you'll have to get all of the current users of it to update their usage–and they'd all rather not, they have other things to do.

If the update is very challenging, developers will intuitively sense that others also won't do it and maybe the update will be canceled.

In the limit with a lot of these, it can make updating your platform like trying to sprint through molasses.

An absurdly challenging coordination problem.

A general rule of thumb: if a thing is expected to change, don't expose it to users whose decisions you don't control.

If the thing isn't changing, then you won't have to do the extremely challenging updates.

If the people who adopt it are part of your organization, you can technically get the CEO to force them to do it.

But if they're members of the ecosystem outside your company, you have significantly less leverage.

How do you know which parts won't change very often?

The safest parts are existing internal APIs that have stayed constant over multiple years, despite new use cases being added in the overall system and underlying changes.

The hardest ones are new APIs that you've never gotten experience with at all.

In these cases, your mental model doesn't have to update much, you just reduce uncertainty.

Sometimes you kick the tires and it's like kicking over a cardboard cutout and discovering a deep foreboding cave behind.

In this case, you must update your priors significantly!

The open and private properties are a bonus, a deal sweetener when picking between two items.

But the ecosystem gives the potential for compounding quality growth.

If users are going to pay for one chatbot subscription, why not pay for the one that has a bonus?

"But why would they do that? Doesn't that erode their advantage?"

"Yes, but they're only thinking 1-ply, and ignoring all of the indirect effects."

It needs an "edge" that draws incremental users into it instead of other options.

By being prominent it becomes a schelling point.

If there is a network effect to the option, then its edge can increase as more people join in, becoming a runaway gravity well effect.

In a truly open system, it's harder for aggregators to emerge.

There's no obvious "edge" to get the compounding loop going from.

When a space is new, being in a tech island with a proprietary advantage lets you move faster and stay ahead of competitor.

The only way to develop and keep that edge is to be proprietary, an island.

But at some point (possibly a long time in the future) the rest of the ecosystem will catch up.

When your previously differentiated proprietary technology becomes a commodity, you'll be trapped.

At this point, it's very very hard to get off the island.

Getting off the island is a massive one-time cost, and that cost rises the longer you stay on the island.

If you do get off the island, you might not make it, and will be actively behind the rest of the ecosystem, at a disadvantage.

Everything will feel weird and foreign and hard to understand.

You'll have to shift your basis of competition to something else.

And there will always be a contingent within the organization that points out that getting off the island will give up your basis of competition and that a better option is to try to regain the advantage by doubling down on the proprietary tech.

But this cannot work.

The non-linear swarm of innovation in the ecosystem will have beaten your linear proprietary investment, and will now dominate.

Still, this argument will be persuasive enough to make the organization conflicted about taking the leap.

Getting off the island will be an important but never urgent decision, so you'll likely wait until staying on the island is an existential, obvious threat, and the danger of making the switch is higher than ever before.

Even if it's extremely important to our lives, like electricity or a fast internet connection.

That is, to verify it does what it says it will do and what the user expects it to do.

Apps, which by necessity have a larger audience, can be audited or verified not to be doing something sketchy by their more savvy users.

But what about an app that was just hallucinated on the spot just for you?

No one else has seen it.

If someone tricked you in creating it, it could do nasty things.

Hallucinated micro-apps need a trust model where the individual micro-app doesn't have to be trusted.

Intuitively users feel that marginal price is fair if it scales roughly with the marginal cost to produce the marginal item.

Fixed cost one-size-fits-all-software has a marginal cost of effectively zero.

That makes that kind of software hard to charge for in the limit.

You can only charge for software that needs constant investment to keep it high-quality, or that gives access to an otherwise inaccessible ecosystem.

The ecosystem is separate from the software, but it keeps the overall value provided by the software fresh.

Aggregators require you to give all your data.. and you also don't get very much in return.

Lowest common denominator, one-sizes-fits-none software.

If you amass a huge audience, the software has to be dumbed down to minimally satisfy everyone.

The tyranny of the marginal user.

Private cloud enclaves allow you to have the security and privacy of on-device... in the cloud.

If it's not resilient to that, the product isn't viable.

Can you make them reusable and safe (to compose, to give access to your data)?

If you're going to leave a horcrux of yourself, you'd damn well better trust the container.

If you're going to build a relationship with a second brain, you have to have deep trust that it will never be taken away from you, and will never be used against you.

Most hallucinated ephemeral apps are not very good.

Conjuring up an ephemeral app covers both fetching one from the ecosystem that's been found to be good in the past, or falling back on hallucinating one on demand.

This allows ratcheting up in quality; LLMs as the floor.

The LLM doesn't have to get it right all the time; it has to get it right some of the time, and then humans in the loop help sift through to find the good ones.

An ecosystem of reusable and safe artifacts.

Ephemeral apps... that you can trust

No idea if this specific example is real, but it tracks!

Any time nature wants to minimize surface area for a volume it reaches for a sphere.

Any time nature wants to maximize surface area within a volume it reaches for a branching fractal.

Extremely efficient to define how to create one given its self-similar shape.

Lots of things in human and biological systems need to maximize their surface area for a given volume

Geoffrey West cataloged in Scale some of the implications of this space-filling shape.