Bits and Bobs 1/19/26

Shocker!

Your intimate friend who knows everything about you and is pretenaturally good at making hyper-convincing arguments being incentivized to get you to buy things is way different from ads in search engine.

It's possible to do this well, but it's a tightrope walk across an alligator pit in a hurricane.

I don't have confidence that OpenAI has the culture or incentives to do this well.

Claude Cowork Exfiltrates Files.

Superhuman AI Exfiltrates Emails.

Prompt injection attacks have morphed into complex, persistent, multi-stage attacks.

Not unlike traditional malware threats.

Prompt injection + malware = promptware.

Who ends up being the 10x vs the 100x return?

It's "who can change how they work."

It only superficially looks like human intelligence.

But when you make it look and sound like a human, it tricks us.

No, they are nothing like us.

They are the aliens in Contact.

The conversion rate for a user actually "installing" one is low.

That's because the actual user experience blows!

This is the kind of thing that only the MBAs in the BD team thinks is a good idea.

MBA's tend to think from the company's needs first.

But what matters is the user's needs.

What MBA's think is cool is a lagging indicator of what's actually cool.

I think both camps are right, but they're talking about two different things.

The value of a model is both its inherent ability (the model quality), and also how easily it can be used to do things (the usefulness of the scaffolding).^[dg][dh]

The raw model quality is clearly plateauing^[di][dj], but there's no end in sight to how much we can do with them, because the quality of scaffolding continues to compound.

You don't need a ton of structure for the LLM to get it.

Humans are way worse at receiving infodumps.

Infodumps are often necessary to communicate sufficient context, but humans can't do it, so we learned to not do it.

But it actually is the best way to give context… if your collaborator can absorb it.

In the code context, this includes "verified it actually compiles."

Since we all think our own slop smells sweet, the person you share it with will almost certainly find it less endearing than you do.

If you couldn't be bothered to read it, then don't bother someone else to read it.

Before slop was a thing, every bit of writing at least someone had bothered to make and thus asserted was worth reading.

But now AI can create content that no one has ever had to assert is actually worth reading.

So it's on the slop creator to vouch that it's worth reading, by actually reading it.

You have to look at the code before wasting someone else's time with it.

We think about algorithmically interesting code as most code.

But that's maybe 1% of code.

Maybe 20% of code is CRUD.

The other 80% of code is just boring integrations.

Convincing someone else to do integration work for you is hard.

LLMs are really good at CRUD code, and also integration code.

Convincing someone to do your bespoke integration work was the main blockage for software.

But now you can execute refactors way faster with the right plan.

You don't need to coordinate the politics of multiple distractible humans with their own incentives because the LLMs will just execute on the plan with infinite patience.

So you get 10x productivity without 10x the coordination cost.

That implies that you need to give them lots of tests.

But often the agents also create the tests.

The agents should own the test set, but the human must own the verification set, and never show it to the LLMs.

Just like in model training!

Will code get so unworkable that it collapses under its own weight?

Or can LLMs successfully continue improving code?

For example, Beads has grown to 248k lines of code.

Is LLM-generated code default-diverging or default-converging?

Different approaches of using LLMs for Elevated Engineering will have different answers.

The race is on to find a robustly default-converging way of working with LLM coding agents.

Both are LLM-assisted engineering.

One is hollow.

The other is resonant.

Compilers were bad at the beginning, but then they got better, and as they did, they gave leverage to everyone who used them.

LLM-assisted engineering is not a passing fad.

If you're running against people on bikes you'll have to give up at some point.

It's participatory creation.

That's a very different kind of thing.

LLMs drop the cost of creation, allowing more participation than ever before.

Blurring the lines between creation and consumption.

You run it until it's done and don't babysit it.

You assume it will converge over enough time on a workable solution.

If it doesn't, that's OK, you didn't waste any of your time.

Just tokens.

How can you make it so that power works for you?

LLM tools are so powerful, but they are hard to integrate into our lives.

Today we've only integrated chatbots into our lives.

But LLMs are so much more powerful than that.

Just because you can doesn't mean you should.

Natural language interfaces are a new party trick.

They are undeniably useful, but the extreme of "everything is just a single natural language interface" is absurd.

Once we as an industry sober up that will become more obvious.

And yet everyone uses the lowest common denominator messaging apps.

When WhatsApp adds a feature, it's so rare and noteworthy that it's on the front page of the NYtimes!

It's that most everyday use cases are surprisingly underserved by normal apps.

That's because most of those features just need integration and that single feature users need.

One of the possibilities unlocked by Infinite Software.

Each meeting is a commitment that is fully "burned in."

That's necessary when you need high precision coordination with others.

But in personal domains, fuzziness is often more useful.

Things need to be in superposition, roughly blocked in.

We hack fuzziness into calendaring software, but imprecisely.

What would a calendar look like designed for the imprecision of our personal lives?

When my kids were newborns I used an app to keep track of when we changed their diaper, when they last fed, etc.

As the kids grew up, I stopped tracking many of those items, but added others.

But the app stays the same, dormant, frozen in time.

What if you could have a personal app for your kid, that grew up with them, and was private just to you?

That's not a community!

A community has two way interaction.

They are where you feel alive.

They often have an in-person component, or some other long-term interaction and trust.

You do that via anonymous aggregation to produce crowdsourced intelligence.

As Tim O'Reilly has said, data is like sand.

Origins get a lot of data, distill it into a processed signal that is mostly anonymous but can add lots of value for the ecosystem.

But the origin has all of the original data, so they will be tempted to peek and look at it in more depth.

What if you got the crowdsourced intelligence but without any goblin hoarding data?

The person could really screw you over in the worst case.

Some collaboration models require a high degree of trust.

But in practice we make trust decisions all the time.

We typically do this for people we know in real life and expect to continue interacting with.

That's much harder in the digital realm.

Still, if you're participating in a cozy community in the digital realm with all people you trust, it's OK if there's some dangerous tail risk, if there's enough productivity upside.

Today you spend most of your time fighting the terrain, not your enemy.

They create positive externalities.

So the question is who has the data to make the best recommendations for you?

That happens when any new input changes.

It's less about LLMs being great at explore (though that's part of it).

It's mainly that LLMs are ushering in a new paradigm.

All of that variation is required to figure out what is worth standardizing on.

The industrial revolution of software is creating an explosion of bespoke things.

So the explosion is the immaturity of the space more than fundamentals of LLMs.

Early users actually do have a literal financial stake in it, so if you're late to the game you're in a much worse position.

This characteristic is often applied to hollow things that are mainly just about speculation and lead to rug pulls, etc.

But what if the same kind of FOMO of shared ownership were applied to a thing that was actually resonant?

It was overtaken by speculation and financialization.

But it has also led to huge amounts of discovery of ideas and building.

Some subset of those ideas will turn out to be extremely useful.

What if you could take the good parts of crypto, throw out the speculation, and keep some of the ideals?

In the future, people wouldn't talk about "blockchain," they'd talk about the thing that embodied those good ideas originally explored in the crypto cambrian explosion.

Where is that soul still alive, as a faint glimmer?

I think the answer is Berkeley.

If AI is a paradigm shift, why waste it by continuing the late stage cynicism of the last decade of tech?

The world is a flurry of distracting activity.

Often the actions take precedence and distract you from your intention.

You lose the plot.

You have to ask it to take a step back and reconsider.

It's so busy taking the next step that it forgets to see where it is.

Humans also do this.

Every hour, ask yourself "have I lost the plot?"

The other approach is to change the game.

We should always ask ourselves, "what's the way to change the game?"

Then a wheel falls off unexpectedly.

The "unexpectedly" is key.

This shifts your priors of how likely other parts that you thought were solid are to actually work.

Now, the expected result is that the plan doesn't converge in sufficient time.

That's a moment to look at the constraints from the balcony again.

Some of them, it will turn out, aren't as important as you originally thought.

Removing a constraint changes the game.

Deciding to remove a constraint typically comes from the outside.

You do it with actual users.

Use cases are guesses at how a tech roadmap will apply to users.

They are guesses in a vacuum.

All that matters is what users do in the wild.

That's the source of the tech industry's focus on getting in the hands of real users as quickly as possible.

If you happen to start at the foot of a small hill, you can quickly hit the ceiling.

It's important to make sure you're at the basecamp of a proper mountain before you start iterating on what users tell you.

An employee will naturally try to balance the long-term, a contractor won't nearly as much.

Renter mindset vs owner mindset.

There are 10^80 atoms in the universe.

There are 10^122 possible games of chess.

If there are runaway combinatorics, you have to figure out how to set constraints to close off vast amounts of territory.

Ideally those constraints cut out mostly the bad while keeping mostly the good.

But if you don't constrain, you'll never even get started.

Our brains actually aren't that general, we strongly bake in assumptions about the world that are nearly always true.

Open-endedness has combinatorics that quickly become overwhelming.

So you need constraints and heuristics that cleanly cut off 99% of the wrong things.

That implies the things you set as your foundations, which provide the constraints, are critical.

Picking good constraints is the secret art of product development.

Cognitive labor will go the same way.

So much of the cognitive labor today is bullshit work that can be automated away.

Many security models require moderation of some form.

The default approaches scale their capability with the number of employees of the company.

But those can quickly get swamped by a compounding ecosystem.

Even better is if there's some way to get some form of moderation to emerge within the community naturally.

That is, that scales with the size of the community.

Apparently it was partially due to TikTok.

But I think there's another inductive engine behind it.

When you hear what it's made of, the default reaction is, "... that doesn't sound very tasty."

Then, when you actually eat it… it's surprisingly good!^[cl]

That diff in the expectations gives you that viral "aha!" moment that you feel compelled to share.

If it didn't sound not very good and then actually taste pretty good then there wouldn't be as much as a baseline drive to share the discovery.

For example, maybe you have a number of bits of functionality spread out across some scripts.

Then you refactor it into one CLI with a consistent UI.

Technically nothing new is now possible… but it is much easier to wrangle the functionality.

Now a lot of outcomes that were previously non-viable (too high of friction) are suddenly viable.

These are the kinds of no-brainer moves that often unlock emergent value, but are hard to motivate with a detailed ROI calculation.

Even if it's only a little bit better, if it has a fundamental bonus (e.g. open, values-aligned) then everyone picks it and it starts developing obvious momentum.

It starts off as a very small quality differential but can then compound into an unstoppable force.

That way when the wind buffets you in a direction you can know if you should fight it or spread your wings.

Prioritization is contextual, not in a vacuum.

So "the wind is currently at my back on this for an external reason" is a great tie-breaker between priorities you care about.

You should prioritize based on bang-for-buck, and the current "buck" has to do with if the wind is at your back.

Don't even think about what to include or exclude, you'll likely get it wrong.

You don't know which parts are load bearing so you just have to assume everything is.^[dl]

Many years ago I heard a legend about Baidu when it first rolled out.

It didn't just look like Google.com's home page, it literally had the same HTML comments in the code.

Comments, of course, don't actually change the behavior… but when they cloned it, they didn't want to break anything that might have mattered.

Premium candles are red all the way through.

It's easier to convince someone to buy a real candle than a hollow one.

That was a deliberate marketing decision.

The web, if it became powerful, would supersede AOL.

So they wanted a generic term that didn't imply the whole open system part.

Chatbot feels like a similar kind of move to me.

LLMs will create a new valuable world-spanning open system, and it will not be a chatbot.

It's clarifying to know who the leader is.

Instead of having to herd cats, there's one person who everyone will follow.

If you trust them to make good calls, that reduces significant anxiety.

In a team it switches from push energy…

…to pull energy.

A milestone is a schelling point created by a leader to cut through coordination cost.

A milestone is an assertion that "this collection of projects add up to more than the sum of their parts."

Giving something a name asserts it's a Thing.

A name is a handle.

It selects into the swirling grey goop of reality and says "this collection of smaller things should be thought of as one thing."

A name asserts "the whole of these things is more important than the parts."

From the individuals and their beliefs.

Power is an emergent force.

A kind of magic.

It's possible if all people in the org have their neck on the line, but that's uncommon.

A single person being sacked is common, a whole team being sacked is much less common.^[cn]

Team members who are at the extremes of a given dimension will feel and carry that tension more than people in the middle.

It's healthy for the team but potentially unsustainable for the individual.

1) Focus on the next breakthrough.

2) Focus on unlocking as much value as possible assuming there are no breakthroughs.

The newer the domain, the more low-hanging fruit there are.

You need all kinds of animals to have a balanced zoo.

Different animals require different habitats.

If you put a conscientious walrus in the savannah habitat you won't realize it's even struggling until it decides to give up.

As in, Myers-Brigg iNtuitive.

Plants are more S

Animals have goals.

They see problems from the balcony and try to come up with strategies to change things.

Plants are on the dance floor, they can only sense the things near them, and need to make do with whatever they're surrounded by.

A mindset: If it's broken, how am I generating that brokenness?

It gives you the curiosity to unpack, and to be in a non-victim / active stance.

Especially important for leaders.

Most of the information that is stressful to you is also wrong or even malicious.

So why not focus on the things that are comfortable for you?

That's a reasonable proxy that feels natural and at least somewhat justified.

But when everyone does that you get self-polarizing, self-intensifying pockets.

All of us in modern society are dealing with the same cacophony.

The things we all do to survive are also inadvertently destroying society.

1) Describing

2) Explaining

3) Predicting

4) Creating^[dn][do]

You have to go through each stage in the funnel in a domain.

Ultimately the only one that actually changes the world is the last one.

All of the earlier stages are just means to that end.

That will isolate you from the rest of the world.

Your success at doing this is tied with the amount of power you wield.

When we're well, we forget how great it is to not be sick.

We just take for granted how great it feels to be well.

When it's broken you realize how great the thing you take for granted.

Same for society.