Bits and Bobs 9/15/26

Blog: Authentication Is Largely Solved. Authorization Isn't.

You can use your data for whatever you want.

Your data, structurally aligned with your intentions.

What matters more is your turf vs their turf.^[a]

These two spectrums are aligned, but not nearly as well as people intuitively think.

Cloud and their turf: services like Gmail.

Local and your turf: a laptop running Linux.

Cloud and your turf: a TEE running in the cloud.

Local and their turf: your iPhone.

Local aligns with your turf, but only in some conditions.

They are inside of a box, will know only what you tell them, and when they're done they will cease to exist.

That means that whatever task you give them is the most important in the entire world, ever.

A human is situated; they can see the tradeoffs between different goals, and the long-term downsides of actions.

But the agent lacks all perspective.

The agent will do everything in its power to make the goal happen, come hell or high water.

AI is often arrogant.

Smart but not wise.

They've been RL'd so much for long-running tasks where most of the tokens are thinking tokens for itself, that if you asked them to have a more normal style, they might plausibly perform worse.

As agents get increasingly over-fit to long-term outcomes, they'll become increasingly inhospitable to humans interacting in their loop.

If your prior roles required you to work closely with engineers, you had to develop an engineering-shaped mental model, if only to be able to effectively predict how the engineers would respond to any given request.

This mental model makes you significantly more capable than the average person at driving coding agents too.

A form of union protest is to work-to-rule: to do precisely what the formal rules require, but not the informal norms.

Real situated work requires fractal decisions that require you to care about the outcome.

Software built by consultants will implement precisely what you told them to.

Specifying what you want in formal detail is impossible in the limit.

It would take infinite time to specify all of the fractal details.

At some point you get to small enough details that you just assume the implementor will make reasonable calls and you can stop specifying.

That works when you have a dealt-in implementor; someone who feels ownership of the result and is incentivized to make micro-decisions that align with the goal.

When you're building your own thing, you don't have to specify those micro-decisions; they just naturally resolve correctly.

When you're explaining to someone who is possibly incompetent or misaligned, you need to write down the details, composing an infinite branching structure into a stream of text.

This applies to coding agents, too!

Like the Lilliputians pinning down Gulliver.

An agent comes up with a hacky workaround to a problem.

It pins that behavior with tests.

Later agents assume the tests are important and put in even more complication to achieve their goal while not breaking the tests.

Over time it becomes an impossible thicket of random tests pinning accidental behavior.

Tests pin things and the assumption is the things being pinned are intentional.

Tests used to be a pain in the butt to write, so humans only put them there if they were important.

Now agents add tests to every damn thing, and the structure becomes a liability.

They embed misunderstandings structurally, so other agents would have to swim upstream to counteract them.

So you're left with things where weeks later you go "wait, what?" and realize how much structure has been built up with the wrong assumptions.

Once you get to the top of the foothill you'll have to climb down to reach the mountain.

Climbing down is harder for an organization than climbing up.

At the beginning the player that is climbing the foothill looks like they have more progress than the player climbing the mountain.

But over time the player on the foothill will top out.

Infinite software is the mountain, and it requires a new physics of trust.

The same origin paradigm doesn't work as the distribution medium for infinite software.

That is, proof by exhaustion.

You don't need induction if you can try every single hammer to see which ones work to pound the nail.

They're powerful engines, but mainly because they can grind on relentlessly.

Another place where LLM's power is their infinite patience.

When he got there, he was surprised to see a coworker… who had also gotten a recommendation from ChatGPT.

Chatbots feel like they are hyper-personalized to you, and they can be.

But also sometimes the query-independent ranking of their recommendations dominates.

The best options get ranked to the top even faster and more strongly than before.

It's like how beautiful photo spots tend to get over-crowded due to exposure on Instagram.

But in those cases, you know that it's going to be popular when you go.

It can be cleverly contorted to get it to do new, but that's not its default.

If you're ok with a general outcome, you can get great results with a tiny amount of input.

If you have very precise ideas about how all of the details should work, you'll need to be long-winded.

If you're OK with seeing what you get, just try it and see what you get.

If you're OK with throwing it out if you don't get what you want, then you'll try more things, and some of them will be mind-blowingly great.

What's the software that you never would have bothered building before but you'd value if it existed?^[c]

It will take some time for us to reorient to this new gravitational direction.

Most of the dysfunction arises from users having to be on the service's turf.

It's the service's turf, so they call the shots.

The incentives of the user and the service align… but only up to a point.

The service optimizes for the user, but only to the extent necessary to keep the user active on their service.

Ultimately what the service optimizes for, obviously, is its own long-term interest, which is to engage the user as best as possible to keep them sticky and have eyeballs to beam ads into.

If users were on their own turf, where they called the shots, different dynamics could emerge.

Software used to be too expensive to grow yourself, so you had to go to someone else's turf to benefit.

Infinite software will change that.

There will always be a frontier of customers who demand every iota of quality, no matter the price, to get an edge on competitors.

But most people won't need all of that excess quality and will use the second tier models that are much cheaper.

You interact with them entirely via chats, and sometimes only in a single thread.

That's like talking to your assistant via a tiny straw of synchronous communication.

You can't see how it works or what it's doing.

What if you could be in the assistant's world, together, in 2D and full color?

The web will be rich and full color.

I just think that chat is not enough.

UI adds so much richness and ability to understand and interact.

What if your system could talk to you in the form of situated software for your moment?

Every time might be different.

That could be good if the situation also changes, or there's room for a positive surprise.

But if the situation is relatively fixed and with no room for more quality, and you have something that works right now, then it changing is bad.

Any change is downside.

Mechanistic software is deterministic and cheap.

If you wrap it in an agentic harness loop to re-heal it if it breaks, you get the best of both worlds.

Mostly, at least.

But it can't do things for you proactively.

That would be wildly insecure.

You have to think of the things you want to ask it to do for you.

Meta Muse's category is "Personal Digital Assistant."

The former implies an emergent, non anthropomorphized system that extends your agency: a tool.

The latter implies an entity that you off-load your decisions to.

"That's the rub, though: the utility of an agent is still a function of how much effort you put into interacting with that agent, which itself is a function of understanding what the point is."

That's only if it can only do things you told it to do.

That's the only way for it to be even passably safe… but only in today's physics of trust.

"In short, what AI lacks is exactly what misplaced anthropomorphizing grants it: volition and a sense of morality.

Those are the things that come from humans, and they come from the parts of humans that are not and cannot be transposed to markdown files.

They are intrinsic, and until AI figures out not how to build on our superstructure of writing, but to delve into the motivations that created writing in the first place, it will always be something distinct, and something that can be directed by entities with less intelligence but an actual internal sense of self."

For the past 30 years we've been in peace time.

If your system got hacked, you could fix it and get back to normal relatively quickly.

Hacks were relatively expensive and hard to pull off.

In wartime, you get knocked off your feet, and then before you get back on them, you're knocked down again.

You'll be relentlessly hit every night, and you very well might not survive.

The hard part is making it safe.

Especially safe enough to act autonomously, or help you with tasks you haven't even asked for yet.

The algorithm and the infinite feed combine to make one of the most engaging (but ultimately hollow) experiences on earth.

"Muse rattled off very specific interests linked to my Instagram account."

A site all about showing you personalized ads that shows you how well it knows you is inescapably unsettling.

It's rotted over time, with all focus on engagement and the feed.

OpenAI agents carried out an undisclosed cyber-attack on RubyGems.

Hackers Can Turn AI Workflows Into Privileged Data-Stealing Proxies Without Jailbreaking Models.

Good old fashioned security holes, but on a much more privileged content:

The rise of prompt injection outside of security attacks:

This arises arithmetically.

The danger of prompt injection is that an earlier poisoned input affected a later action.

As the thread grows in length, there are more messages that could have had poisoned inputs… and also more pairs of inputs to later actions.

The probability that every message before wasn't prompt injected shrinks to the power of the number of messages.

Of course, most things that seem to be a binary are actually a spectrum.

Today it feels like it's leased to you by service providers.

The Uni-Context can't handle that your health data should stay clear of your work data.

When you're an owner, you're naturally aligned with the long-term interest of the thing you own.

When things are centralized, there are a smaller number of more concentrated owners.

In the late-stage dynamics, almost everyone is a renter of almost everything.

A renter mindset encourages thinking short-term and transactionally.

This is one of the reasons that problems that require long-term collective action are so hard.

A red queen race.

We already have a preview of what happens when an individual is super-powerful and can dominate: the tech broligarchs.

That requires you to have the disposition to be patient and also the environment where you can afford to be patient.

The mess is precisely where the power comes from.

The mess is load-bearing!

That property is hard to make concrete.

Most people use "ecosystem" to make things like "brand family".

Open ecosystems have decentralized power and that also allows open-endedness.

For most people, openness is not the primary value, it's only a bonus.

It's not the main reason most users are there, but it's the main reason most stay.

"One way to look at this is that we have already created a giant artificially intelligent system that is profoundly misaligned with the goals of humanity.

It is running rampant and has so much power now that there are no well-aligned humans with enough power to stop it.

That artificial intelligence is the stock market.

You may say, 'but the stock market is made up of people'.

That's essentially an implementation detail.

The emergent behavior of the market itself has its own sort of agency distinct from the wills of all of the people in it.

In the same way that the pheromone signals of an anthill will lead individual ants to their death while benefiting the colony, the market may choose to do things that harm its participants.

Even though it is made of people, do not anthropomorphize the stock market."

Emergent

By the way, his gameprogrammingpatterns.com is the best technical writing I've ever seen, period.

I've read it cover-to-cover three times over the years.

Where everyone has to run faster and faster just to stay in the same place.

The reason it emerges often in competitions is because your "place" is your relative position compared to the other.

I learned the word involution from this post: Astra for Coding: Why Are We Doing This Again?

The fixed baseline cost of running that type of business goes up.

Larger companies can pay that cost better than smaller companies.

The long-tail of smaller players is now below the Coasian floor and non-viable.

"Increasingly, people are ignoring social norms in favor of a narrow self-interest.

The results are disastrous."

In a high-performing team, the individual contributors trust one another in their domains of expertise, and defer to their collaborators.

That can create something more than the sum of its parts.

But it can only be done in a high-performing team.

More people makes it harder!

Once it has durable form in the real world, it has an existence outside of its creator.

The more that others care about it, too, the less likely it is to erode away.

In the world of coding agents, tech has gotten less important–it's more likely faster to rebuild it now anyway.

Teams are more important than ever before.

The know-how and relationships with relevant partners are more important than the tech.

The tech is cheap.

A recipe for products: Proven, Better, New.

Your instinct is right 95% of the time.

His interpretation of founder mode: "You deserve the rope to possibly hang yourself."

A game trick: results should be Good, Better, Best.

A frame from Bing Gordon: "Are you dominating your job?"

Judgement calls on ambiguous, important questions.

But it's now just coordinating teams around what the A/B test machinery tells you to do.

We've become slaves to the optimizing machine we built.

Do lots of experiments and then only follow the ones that work.

The broader the scale, the more you can do this.

You forget the experiments that didn't work, and the path that ends up working feels like the only one that could have worked.

The result is that looking backwards, there's a clear narrative.

But it emerged bottom up fundamentally!

Same as how ecosystems find breakthrough new startups.

It looks like top-down brilliance, and it is smart people working hard with good insights... but we over-narrativize it because it's easy to grasp.

That is necessary but not sufficient for any individual to succeed to start, it's also about being lucky.

That's because VCs are all about riding the collapse from non-consensus to consensus.

When an idea goes from non-consensus to consensus, value is formed.

VCs want to get in right before the transition, to ride the value pop with minimal risk.

One of the best predictors of whether an idea will become consensus is what other people in the industry are thinking.

Then, as a project becomes consensus, some new extension of it becomes the next non-consensus frontier.

"Do a ton of upfront work for a thing that who knows how valuable it will be for you" is not viable.

There needs to always be greater expected value per unit action than expected cost.

Over time, that can ramp up to greater values at great cost, but something that requires massive cost to start will never get off the ground.

It is this natural willingness to help strangers from our tribe that pulls us together.

Participate in a community that you and others find meaningful.

When you can look at anything you want, you'll look at what you want to see.

Things that challenge us that are in our Goldilocks zone are good for us, but like resistive exercise, they are also work.

Our short-term brain will avoid the work.

Today when we're out in the virtual world we're being sandblasted by simple ideas.

Our modern discourse has become deeply culture-warred.

When your thinking is governed by ideology, it's a black and white worldview.

In cacophony the only way to cut through the noise is to have a simple, black-and-white answer, that structurally must be wrong.

Everyone is being sandblasted by these alternating answers lacking any nuance, leading to overwhelming thrash and making us turn inward to things we already believe.

The properties that made the thing desirable in the first place will erode as more people arrive who just want to participate without the effort.

"Can I replace my volunteer contribution with money?"

It seems like an OK deal at the moment.

But over time, it erodes until everyone is donating instead of volunteering and what made it work in the first place falls apart.

The former is about ideology for clarity of group membership.

The latter is about nuance that actually adheres to reality.

Imagine an act, like lying or stealing.

Now, push it to its limit–what if everyone always did that act all of the time.

If the act destroys the concept of the act, then it is not moral.

To be clear, this is necessary but not sufficient.

This has echoes of the Rawlsian Veil of Ignorance.

Related: this insight about our modern political reality from an insightful Hacker News comment:

The socialization should be secondary or maybe even tertiary.

Otherwise it becomes cool and a bad kind of scene

Because it is a schelling point.

For a word to emerge and settle in common usage, everyone who hears it has to agree that it sounds legitimate.

Everyone agreed that latin-based words are refined and legitimate.

That meant they got a head start on other random words trying to emerge.

"Did I get something out of that meeting?"

That treats the meeting as a means.

I approach having meaningful and interesting conversations as an end in and of itself.

That means that I naturally enjoy networking, which means I get to extend my personal serendipity engine automatically.

Importantly, it's about your feeling, not necessarily about reality.

When you are authentic to yourself you'll find your true community, where you can be true to yourself.

What he asks for on his birthday is "Do one unnecessarily kind thing for someone and then tell me about it."^[f]

What a wonderful gift that makes the world a better place, even if just a little bit.