Bits and Bobs 5/12/25

Technology that is perfectly aligned with my intentions.

Not optimizing for anyone else, (especially a corporation), but for me.

Not necessarily what I do but what I intend to do.

My higher aspirations, not the engagement traps I fall into.

No one intends to get stuck in an addictive doom scroll loop.

Too much technology built by companies today is happy to get you into an engagement loop they can juice for ad revenue.

Intentional Tech is of critical importance in the era of AI.

Normal programs can't write a sonnet to save its life.

LLMs can't give you the same results repeatedly to save its life.

Deterministic vs non-deterministic computers have different strengths.

System 1: powerful and deterministic but finicky.

System 2: broad and stochastic but forgiving.

Humans have Systems 1 and 2, and so do computers now.

The future is obviously the combination of both system 1 and system 2, not either or.

Maximally using LLMs will require context and tokens.

Focus on the users who are living in the future, and make them successful.

Chatbots are great for unstructured tasks but can't do structured well.

To help with orchestrating our lives, LLM-powered tools will need more structure.

Think of it as a "vertical OS" for parents of newborns.

BabyConnect is not special; there are dozens of similar apps.

It's basically just a handful of CRUD UIs on top of a SQLite database specialized for parents of newborns.

There is absolutely nothing special in the app, but it's still indispensable.

This app could go away at any moment.

There was no good alternative:

Imagine how many other little niche vertical OS style use cases exist that are below the coasian floor.

Where a simple CRUD on top of a spreadsheet data would be life changing.

Getting Things Done is a powerful process to be more productive… but it takes a lot of learning about it and discipline to apply?

What if just talking to a system aligned with you would naturally help you get things done?

If you had a coactive system, it could help you get things done automatically without having to ever know about the formal Getting Things Done process.

The key insight: a document is a great medium for collecting unstructured data.

Imbuing a document with even small amounts of mechanistic magic can make the experience feel radically more productive.

Instead of applying heavy-weight software, the content in your document just magically comes alive and more functional as you use it.

It shows the power of a coactive medium for getting things done.

Imagine what you could do with that kind of power not just for travel use cases!

What if it could show you exactly the affordances aligned with your intentions in that moment?

You'd need software that could self-assemble.

They are self-assembling software.

They help you solve problems, as an extension of you and your intentions

Their lack of structure helps you get started, but prevents you from making progress.

Chatbots are the faster horse.

As an industry we're so distracted by Chatbots.

Chatbots are the most obvious use of LLMs, what you'd come up with after thinking for literally 30 seconds.

Their obviousness is like a bright light, blinding us to everything else.

Chats are flexible enough to get started with anything.

But they are the wrong UX for long lived tasks that need more structure.

We've missed that they can execute basic tasks on basic substrates very well.

LLMs create the possibility for coactive software.

It's totally invisible to us because we don't realize it could ever be different!

Orchestration doesn't necessarily mean doing anything, but rather keeping track of all of the threads of execution in your life.

Orchestrating all of your relevant context is a black hole of time.

Orchestration is highly contextual.

To do orchestration requires integration.

Also when it does make mistakes you can't introspect them.

That means it needs to hit 99.999% accuracy.

It's easier to hit that bar with deterministic things.

Very hard to hit it with non-deterministic things.

Information is only context if it's contextually appropriate.

The wrong information is noise.

If you say "we'll have your context" you hand wave over the hardest part of it--curating the right context for a given situation.

Undifferentiated stuff.

But not all content is the same.

Not all context is the same.

The original Google Now was wonderful.

The actual features were mostly 20 or so simple hand-created little recipes for UX and when to trigger.

The magic was just the context.

If you already have all of the other adjacent context in one place and up to date, it gets easier and more valuable to add each incremental piece of context.

This is especially true if you have to organize your context for your family where you have to share tons of potentially sensitive information with your partner.

So there's a strong pull to put more and more structure and data into your system.

But the more structure, the more manual effort it takes to maintain and implement that structure..

The more effort it takes, the more likely you get behind.

The more you get behind, the more likely you get very behind.

When you get very behind, the more likely you are to call bankruptcy on the whole system.

All but the most disciplined people will at some point inevitably stop using their orchestration system, after having sunk huge amounts of time and effort on it.

The reason for this diversion is that humans are responsible for all of the mundane, mechanistic effort.

An insight from a friend: "People don't want a better Notion, they want a librarian."

The tinder swipe mechanic gives you the feeling of getting things done, oversight of the updated information, and also it shows you what it's doing for you.

When I had an extra 15 seconds I might spend time on that instead of doom scrolling.

Having one centralized relationship with AI doesn't even make sense.

That doesn't work!

We contain multitudes, we show up in every context differently.

It only knows what I told it (which might be a weird partial subset).

But giving it more information is creepy.

Where do I dump my life context in a way that will allow LLMs to work on it?

A trusted place just for me, totally aligned with my interest.

I could utter a single word to my husband that would require me writing a book for someone else to understand.

Context is about rich, nuanced understanding of the particular details that matter.

Context is the key for unlocking particular meaning in a given environment.

I think this take is correct in some ways but incorrect in others.

A system that works entirely for a user, that they pay for, and that is entirely private to them, and acts as an extension of their agency doesn't have that problem.

The problem is not the context and personalization, the problem is the alignment with a user's agency and intentions.

Personalization is useful, it's just that today it requires the faustian bargain of giving up your data to another entity with ulterior motives.

That's how it works today, but that's not how it has to work.

But that means your Private Intelligence needs to be totally aligned with your intention.

A race by the aggregators to capture as much of users' context as they can.

They're all trying to build a walled garden larger than any that ever came before.

Their product choices are about getting more context.

Corporations are salivating over the user context prize.

Fracking is not good for people in the long run.

Related to Sam Lessin's notion of AI fracking content.

If your context is locked to one model then you can't swap them out, and then you can't try other ones

The risk of a monopoly of models and services: a single world view that everyone is pulled towards, intentionally or unintentionally.

Why might context portability happen now when it didn't before?

A dossier is not about understanding you, it's about making you understandable to a bureaucracy.

A dossier is context someone else maintains about you.

It's about distilling the key, sensitive data to make sense of you to someone or something that doesn't know you.

The word "dossier" implies something clandestine and nefarious, not aligned with the user's interest.

Dossier: a deep thing about you that has power that you'll never be able to see.

This week I learned that apparently part of the motivation for laws like HIPPA was a case where a given person was denied a university position based on a detail in their packet that was factually incorrect.

Had they been able to see it, they could have pointed out the error.

A prompt to get ChatGPT to divulge the dossier it has on you:

Your dossier includes things like "9% of the last interactions the user had were bad".

It presumably could include things like "The user is insecure about people thinking they're not smart enough."

Prompt injection with tools that might do network effects could leak significant facts about you!

What that means is private to me only.

Totally aligned with my interest.

If it's not truly personal, the more powerful + proactive it is, the more terrifying it is!

Power that's misaligned with my incentives is scary.

Imagine how much more comforting it would be if a company could say: "Not only will we never sell your data, but we can't even see it in the first place."

You wouldn't want to have your therapist know about how you raided the fridge last night to eat a slice of cake.

Or imagine a system that you tell your deepest, darkest desires to… that might accidentally divulge some of that when you interact with it in front of your boss.

The contexts are separate!

Having them all mixed together is potentially explosive.

Sometimes you're in goofy pelican mode, sometimes you're in serious mode.

If the system has a personality you have to reason about things like:

If the powerful AI system has its own personality, then it could dominate mine.

"I can't do that, Dave."

Chilling!

If they know your context, they can do a very believable job of confirming your bias.

A TV channel perfectly tuned for just you.

Amusing ourselves to death.

The top 4 chatbots today are led by people who have been Facebook execs.

OpenAI is speedrunning the engagement maxing playbook.

The engagement maxing playbook was a net negative for society on its own, and now we're supercharging it with AI.

Imagine a sycophant-on-demand that is created by a company that wants you addicted so they can show you ads.

Terrifying!

Your context is the most important animating force.

It's trapped in random cages.

Your context can be used to help you... or manipulate you.

Like the other kind of C4, this problem is explosive.

Personalization is not the problem.

It's the corporation doing it on your behalf that's the creepy part.

To do it correctly requires a system that is human-focused, not corporation focused.

Your brain's background processes chewing on your tasks and making connections even without you being consciously aware.

Your system 2 is connecting ideas even when you aren't paying attention.

What if we could have an offboard System 2 to chew on these insights for us?

Today to make a computer do what you want it to, the user has to be managing the context and orchestrating--which takes a huge amount of mental effort and focus.

What if you had an omnipresent little container that you could just speak or drop something into and it filed it away and made the connections for you--whether it was a deep insight, a tactical reminder for a few minutes from now, a gift idea for your spouse, etc.

A coactive tool for thought.

Extends our neocortex: an exocortex.

Decentralization is not the end, it is the means.

Cognitive Liberty is the end.

If you have an exocortex, it is critical that it belongs to you and is aligned with your intention.

Mass-produced software operates at a scale where there's no other way.

The web, for example, is a medium.

Mediums are about integration between disparate things into one emergent whole.

Whereas the benefits are all direct.

The swarm follows direct, not indirect incentives.

So everything gets more and more centralized, which harms adaptability and resilience, and centralizes power.

Centralized power is corrupting.^[na]

Computation is like alchemy; it should be used to extend our agency.

Give the aggregators our most precious context and they give us free features that make our lizard brains happy.

Tumbling down the engagement-maximizing, meaning-destroying gravity well.

The aggregators have sucked up all the oxygen.

They control the distribution and the attention.

Anything that challenges them doesn't even get to take its first breath.

AI could either usher in the enlightenment, or push us deeper into the dark.

Succesionism is about building a worthy successor "species".

Hyper-engagement maximalist is a cynical business ploy.

What about a human-centric vision of flourishing in the era of AI?

Different people would want different things, but what's important is that everyone is living more aligned with their aspirations.

AI slop can be valuable if there's a human you trust endorsing it.

Among the sea of slop, a thing that someone you trust endorsed can stand out.

There are diamonds in the rough, if someone can point them out to you!

"Through Humboldt's lens, the work we choose defines us. Not just as economic beings seeking survival or material comfort, but as the architects of our own becoming.

As humans we arrive with innate potentialities: latent capacities and natural inclinations that provide starting points for development. It is very often through our work that we discover these potentialities, develop them through practice, and determine how best to express them.

Humboldt recognized a fundamental tension in his age that has only intensified today: when systems promise efficiency and optimization of our path, they risk diminishing our capacity for self-authorship."

It was an annotated copy of the 10k lines of Unix source code back in the 70's.

Apparently it was a highly pirated book--only people with a license to Unix were supposed to be able to see it.

The core 10k lines describe the elegant physics of the system and the three fundamental "particles":

That's it! Out of those ideas you can get a universe of amazing things.

The combinatorial power of those primitives also sets a ceiling of what is possible.

Basically every computing system we've used for decades uses these fundamental particles.

What other universes are possible?

This frame comes from The Heart of Innovation.

Informative: incrementally extend what's already there.

Transformative: change the game of what's already there.

Formative: create something new.^[nc]

Informative innovation assumes the structure is roughly correct and it just needs to be optimized or tightened.

Transformative innovation assumes the structure must be changed.

If you want to change the world but do not have leverage over a system, you must do formative innovation.

Formative innovation must start small, as a little demon seed.

Over rotating on either is dangerous.

Either you get lost in the Xanadu of your imagination or you get overly constrained by reality and don't change it.

When you're trying to change the world with some formative new technology, it's easy to get lost in research land and lose touch with the real world.

Urgent tasks are obvious.

Important tasks are often not obvious.

But it's harder to grab onto.

So people don't.

They focus on the obvious not the important.

That is, things that once they exist are obviously better.

An example: people needing to cross a river to get to work.

This frame also comes from The Heart of Innovation.

The "not not" frame helps clarify indirect value.

Most other frames focus only on direct value.

An entrepreneur sees what they think to be a means.

If it doesn't work it doesn't matter.

Entrepreneurs constantly seek disconfirming evidence.

In idea space everything works exactly as you expect.

Because it's not real, it's your simulation of reality.

You don't actually want disconfirming evidence so you don't get it.

Disconfirming evidence must come from outside your mental model, ^[nf]because by definition everything in your mental model is confirming of the mental model.

The real world doesn't care about your idea so it ruthlessly generates disconfirming evidence.

Staying in idea space feels good because you feel like you're solving problems but in reality you're just generating more confirming evidence.

If you have two pace layers intermixed, they fight each other in an eddy current and neither can run at their fastest speed.

When you split them apart they can go faster at their natural pace.

Smooth is fast.

Laminar flow is orders of magnitude faster and easier than turbulent flow.

Communism doesn't work because it requires a top down, omniscient administrator, which obviously doesn't work.

Capitalism is all about "that's impossible to coordinate at the society level so just have a swarm, and make sure the natural incentive is to provide value for others."

But then within capitalism companies are often run like communism: command and control with an implicit administrator.

Why doesn't that obviously not fit?

Perhaps it's about the Conservation of Centralization.

On one side of the boundary it's bottom up so that means on the other side it gets net more top down to compensate.

If everything were bottom up, everything would be chaos

If everything were top down, it would be extremely fragile.

Easier to control.

Efficient.

But they are much less resilient.

Less likely to have great results.

You might experience the company as jerking you around capriciously.

But the company is actually a swarm.

More like a swarm of bees with a sheet draped over them.

It doesn't have a brain, it is an emergent swarm.

It doesn't have its own goals, it does not even know who you are.

If it's a coach or mentor it's encouraging: "you can do it!"

If it's a manager it's discouraging: "even if you manage to do this thing, you won't get credit for how hard it is."

The superficial trappings of quality, not the fundamentals.

An interesting paper that examines this: Pareidolic Illusions of Meaning: ChatGPT Pseudolaw and the Triumph of Form over Substance.

Some people gleefully ski down it.

"Well I'm in this race, I might as well win it!"

"... But it's a race to the bottom!"

Transparency doesn't help you avoid gravity wells.

Everyone falls into gravity wells by default.

Escaping a gravity well requires some source of compounding energy to fight getting pulled in.

A computer with a single light sensor is dumb and blind, obviously.

You realize that when you try to program it to do useful things in its environment.

And yet we're more like that then we realize.

Our Umwelt is rich, but still missing signals, like magnetism, a rich sense of smell, etc... and other signals we can't even imagine.

A smuggled infinity narrative is useful to get coordination on big projects.

Even if the vision is impossible because it has a smuggled infinity, it still does align a lot of disparate actors and allows building things that wouldn't be possible without that alignment.

A useful, if chaotic, alignment mechanism.^[ni]

This is one of the main points of Byrne Hobart's Boom: Bubbles and the End of Stagnation

It would be not about replacing humans, but about extending our agency in prosocial ways.