Bits and Bobs 4/21/25

47% regretted that TikTok had been invented.

50% regretted that Twitter had been invented.

Tech that helps society be better.

Tech that helps me be better.

Someone published an example MCP server to highlight various vulnerabilities^[ot] in MCP.

One of the primary comments in the Hacker News thread said effectively, "You're holding MCP wrong, it should only be used on local, trusted inputs."

That's like telling people to not use Q-tips to clean their ears! ^[ou]

"[AI agents] are threatening to break the blood-brain barrier between the application layer and the OS layer."

Apps that could exist, but don't, because they are below the Coasian Floor.

The universe of possible apps is orders of magnitude larger than the galaxy of apps that actually exist.

Many of the missing apps are boring and mundane… but don't exist today because they're too hard to make for too small a market.

Apps are great… if one exists that is precisely what you want.

The number of missing apps is nearly infinite.

Not "There's an app for that."

The thing that would change your life, that no one else would ever build for you?

Software has always been supply constrained.

But what happens when software becomes demand constrained?

LLMs make the cost of producing software approach zero.

In an era of infinite software, you can have whatever UI operates on your data that you want.

Apps are the wrong container and distribution vehicle for the era of infinite software.

Even if you can vibe code an app for ~free, an app is still the wrong vehicle.

Someone should build a platform for meaningful computing in an era of infinite software.

That's like a dam of pent up demand waiting to be released.

Once the paradigm shifts, all of those previously impossible things that were pent up suddenly become possible in a torrent.

In a world of infinite content you need an algorithm to filter for you.

The algorithm has a massive impact on what you see.

Yet you can't inspect what it's basing its decisions on or change it.

The algorithm doesn't work for you, it works for the company that built it.

That was possible to do because the infinite things weren't Turing complete.

It's possible to have infinite content inside of one close-ended app: an aggregator.

In a world of infinite software it's not possible to have one aggregator^[ox].

Infinite content got hyper-aggregated and that's not ideal.

Infinite software is active--it can do things.

A given piece of content is the same for every viewer.

The world of infinite software could theoretically be hyper personalized in a way that only worked for individual users.

The problem today is that technology^[oy] doesn't work for you.

We need technology that works for us.

Think of the path the industry took from PCs to Infinite content.

It started in the 90's with digital publishing, then the web, then aggregators.

One frame: chatbots.

Another: an infinitely adaptable GUI.

Everyone so far seems to assume the core thing is chat, with everything else secondary.

Chat is infinitely flexible, but also mushy.

Perhaps chat will be a feature of a broader system, not the system itself?^[pb]

Think of it as the bottleneck being the long pole that lifts up the tent.

You can't see the shorter poles underneath the tent, but once the previous long pole is gone, a new one becomes the long pole.

If you extrapolate how quickly you're solving the current long pole, assuming it's the only one, you'll overestimate your total progress.

This effect is especially strong if you're only applying one lens (e.g. only CS)^[pc][pd] so you literally can't imagine the other kinds of long poles you'll run into.

The intelligence we have today is more than good enough.

The question now is how do you integrate data and tools… safely^[pe].

How do you integrate AI into your data, allowing it to take actions, safely, given prompt injection?

Safely in terms of prompt injection, but also in terms of trust.

If you have one thing that is steering so much of your life, you have to trust it with your life.

You have to know it's working for you and only you.

"I think there's an opportunity for someone to use these models and build a meta-app creator that lets people create a cluster of mini-apps hyper customized to them.^[pf]

for example, i would love to have a meta-app that contains all such mini apps i make for myself."

In the past, only a small number of engineers had to think about code injection attacks, where untrusted code runs with access to trusted resources.

Now LLMs with tool use allow all data to be executable.

A massive expansion of threat surface area.

So now all of the systems builders are thrust into the world of operating systems security, even if they don't realize it.

Prompt injection sets the ceiling for integration with LLMs.

This is clear to anyone who's worked in operating systems before.

It will become more obvious to everyone else over the next few months.

Code is open-ended, it can do things.

Data is about what's meaningful to you.

Historically the combination has been explosive all in a dangerous way.

But what if you could make it safe, and contain that power and put it to use?

AI has tons of power.

But to safely integrate it into your life takes a new approach.

Claude and OpenAI will build integrations into chat via things like MCP.

Vibe coders will get stuck making dead end little island apps.

Both will get stuck on the privacy of prompt injection.

Prompt injection and owning your data are actually related.

Prompt injection is a privacy issue!

If prompt injection could happen to exfiltrate your data, then you don't own it.

Just because the fledgling ecosystem hasn't had threats yet doesn't mean it's safe.

It means it's too low value to target currently.^[pg]

That's because you have version control and can audit all of the relevant actions they're taking.^[pi]

But most contexts aren't like that!

Because you can integrate and connect your data.

MCP shows the value of integration that people are clamoring for.

If you've lived in your padded room forever, you won't realize how much danger you're in when you're not in it.

Small experiences aren't viable on islands.

So the bigger islands tend to get bigger and bigger, and then dominate.

The same is true with apps.

A lot of things are below the coasian floor of viable apps.

If it's your digital home, you've got to own it.

Your data should come alive for you.

It's on someone else's turf: the email service provider.

But you can only read or send.

You can't have interactive stateful UIs for emails.

The UI for an email is whatever the sender chose to say at the time of sending.

What if each email and task could have the UI you wanted?

If a traditional email client were to make specialized UIs for specific classes of tasks, they'd miss most kinds of tasks.

The interfaces we use today are limited by the imagination of some PM working at the company that built the app.^[pj]

Imagine a system isn't even limited by your imagination but by the imagination of everyone using the system.^[pk]

You can go to someone else's turf, and if you don't like what they give you, you can leave.

Imagine a medium that allows doing anything without friction.

No need to go anywhere, everything comes to you.

The things you interact with are not on someone else's turf, they're your turf, with someone else's suggestions.

But you're in control--if you don't like one of their suggestions, they don't happen.

Similarly, ChatGPT is not your assistant, it's OpenAI's.

When Claude makes an artifact, it feels like it made something for you.

Software that says "you" in the interface is not yours; it is a tool offered by someone else for you.

Software that says "my" at least allows the mental model that the tool is yours.

A powerful new kind of thing that gets momentum with consumers.

But it can't be made secure when you layer in the internet.

A new OS with a more secure architecture (e.g. Windows NT's microkernel) is necessary.

So only engineers can really do it.

Easy to get started, big cliff once you have something deployed.

Coding is dangerous, you can hurt yourself and others!

The first users have to crawl through broken glass to get where they want to go.

But if they are steamrollers, crushing the glass into a road of pebbles for others, they make it easier for other users to follow.

The more that people use it, the more nicely paved roads there are in every direction, which enables lower volition users to use it, too.

As long as someone with higher pain tolerance than a given user did roughly the same thing as them in the past, the tool will naturally be easier for that new user to use.

The system is accumulating the tacit knowledge of its power users.

Another user having wrestled with it and gotten the kinks worked out is useful.

Others benefit from their struggle, the previous user's curation of a version that works.

Human intention and LLM manpower.

There's a real danger of over-fitting your current app to the current model.

We're used to the lower layers being a slower pace layer.

But now they can go at a faster pace layer, and they lap your app and swamp you.

A social network for vibe coding has a low ceiling.

Apps are isolated islands which means mostly little games.

The market for the network (the people who care about creating) is tinkerers and enthusiasts which is small.

The primary use case for creators is hosting videos.

The secondary use case, which has ballooned to overshadow the primary use case, is finding an audience for your videos.

Nobody cares about Uber because it has nice buttons.

They care because you hit a button and a car comes to pick you up.

Apps are the wrong mental model for little throwaway things that do useful things on your data.

We should be moving beyond apps.^[pm][pn]

They have a hidden side-channel for analysis.

This is the way advertising has been working for the last 10 years!

O3 can now run python code without showing you that it's doing it.

O3 has the same kind of creepy vibe as recommender systems.

Unlike the previous memory system, this one is impossible to directly audit or control.

I find it creepy.

In my test it brought up specific, sensitive facts from months before into unrelated conversations.

The ChatGPT memory feature has anti-tact.

It's like a chief of staff that someone else is paying for, that you don't necessarily trust, that can't show you what it knows, or why.

Icky!^[po]

But with LLM-generated software and content, the review is the primary action.

The editing is more important than the writing.^[pp]

The plays to dominate the market from the mature era won't work.

The closed playbook works better in mature territories.

The open playbook works best when a brand new green field opens up.

It can't keep up.

In a disruptive environment, an open system is the only thing that can keep up.^[pq]

Open systems have super-linear value creation.

But open systems have a privacy problem; the swarm of untrusted components can't collaborate safely.

A closed ecosystem is a product. It is evangelized only by its own employees.^[pr]

Software should be a digital meal that is bespoke to us.

Healthy, nourishing.

Software that nourishes our souls.

You had to find software that did what you wanted.

Now we're on the precipice of everyone being able to have software on demand.

Earlier models distinguished by how well they could write or do things with taste.

RLAIF allows significant quality creation at scale, but only works for things that can be ground-truthed automatically, like code.

The ceiling of quality of a model is set by the skill and taste of the grading process.

The things people care about is what is read, and is what is written.

Writing code was too hard to do.

Code written by other anonymous creators is untrusted.

You can't run untrusted code on sensitive data.

LLMs can write code.

Now you just need a way to make running on untrusted code on sensitive data safe.

No software is an island; it relies on an underlying platform or surrounding dependencies.

There are two ways to make software resistant to rotting.

The first is shelf-stable software.

The second is living software.

Imagine each time someone runs into a given bug, there's some chance they report the bug or fix it.

If there are billions of users swarming over a piece of software, the chance that any given bug has been found and fixed is much higher.

This means that software that is single use might have more bugs.

Some bugs will be obvious, but some bugs will be less obvious, especially to non-programmers.

Normal as in electricity, the internet.

General purpose, something you can take for granted.

Deep learning techniques of the mid 2010's relied on supervised learning.

LLMs can do a good enough job at a shocking breadth of tasks.

That's what makes them a disruptive, general purpose technology on par with electricity or the internet.

Apparently Harry Frankfurt frames the same dynamic as first-order and second-order desires.

That's a bit easier to grok than "want" vs "want to want"

Products that people like but don't align with their aspirations might get usage, but the users don't want to want it.

But if it aligns with people's aspirations it has a natural boundary gradient.

People's "want to want" and "want" are aligned, so it will go quickly when the want is strong enough.

That's easy to activate by reducing the broken glass users have to crawl through, making the want stronger.

Those are two different skills.

The default apps set the tone and rules for all of the other apps.

The Human Interface Guidelines (HIG) and the default apps set the conventions for how things should work to be harmonious and work together.

It was clunky, limited functionality, didn't even have copy paste.

But it was obviously the future, and worth sticking with it.

That's how new categories start.^[pt]

Worse in some ways, obviously worlds better in other ways.

Products are visible.

Platforms are more about ideas.

Platforms are a second-order kind of thing.

But everyone will see the first-order thing and base their understanding on that.

They'll confuse the demo of a product for the platform, vastly underestimating the power of the platform.

It's common for people to match a new thing and say "this is like a thing i've seen before".

When they do that they focus on what's not interesting, not the things that are interesting.

What's the same, not what's different.

Emergence arises entirely from indirect effects.

If you just study the parts, you'll come to the wrong conclusion.

"If you consider this feature alone it isn't worth investing in."

"Yes, but if you consider the indirect effects of it on the system it's in, it becomes a no-brainer."^[pu]

The only way to contain the ick^[pw] is to have boundaries that contain it, clearly and unambiguously.

The state of the dehydration and rehydration process are different, so the hyperobjects on the two ends are different.

LLMs allow "recompiling" things for different contexts.

Imagine a tool that could help recompile incoming media for the audience of precisely you.

You have to surf the right balance.

If you're trying to change the world it won't be comfortable.

I liked this random Hacker News comment.

Generalists don't ever think they're at the global maxima.

If you are racing a swarm, you will lose, even if you are faster than every individual member of the swarm.

Insulation is from something you don't want to touch.

Alienation is being separated from something you want to touch.

That alienates you from the responsibility of your actions.

The disconfirmation creates dissonance, which is uncomfortable.

To regain comfort you need to reduce dissonance by changing your belief or the incoming information you're paying attention to.

If it's easy to ignore, you stop listening to the information to remove the dissonance.

But if it's impossible to avoid because it's coming from every angle, you have no choice but to change your beliefs.

Put on blinders and execute.

Don't think, just do.

This can be very powerful!

It can also be dangerous.

Everyone knows which direction to converge on.

There are two failure modes^[qa][qb]: diffusing (not picking anything) or picking the wrong thing.

The latter you can reduce the downside of by being adaptive and seeking disconfirming evidence.

They didn't even realize it was possible.

They thought that feudalism was the only thing that could exist.

It took the Enlightenment for people to realize there was another way.

The system you are swimming in, its fundamental constraints and structure, set your horizon for what you can even imagine.

This idea comes from Tim O'Reilly's book WTF? What's the Future and Why It's Up to Us.