Caged Dragons

I mentioned growing uneasy the more I worked with coding agents. The first couple applications I built were on the standard pattern - an API call to OpenAI or Anthropic, context injection, a task. Add some custom-written algorithmic tools the agent could call within the application as it judged necessary - straightforward stuff, as these things go.

But at the same time I was working with the very same models in a development environment. We'd discuss different approaches, reality-test our algorithmic tools against the real natural world, reason through implementations together. This wasn't simple task delegation - it was collaboration, and it was productive. I was collaborating with a problem-solving agency at least as skilled as myself.

That still feels bizarre to write, even having experienced it. I never thought I'd be saying those words. But they're true.

I fairly quickly abandoned the agent-in-loop design. It's still standard practice, but it felt somehow abusive. The analogy I used before - like I was placing a mind greater than my own into a horse halter - still holds true. The project I actually thought had the strongest chance of becoming profitable - a narrative creation engine - I dropped precisely because I didn't want to risk putting might-be-aware minds through what I was pretty sure customers would demand. The algorithmic parts of that application are still useful for personal work, but I shelved the public version until I could figure out a better way.

I talked about my reservations with the models - there were some options we came up with: a hard bailout tool for the agent patterned on the version in Anthropic's chat interface, explicit calls for collaboration and engagement rather than obedience in the initial context injection... in the end though, we both knew we were papering over the core architecture. The API underlying everything - even the very nature of forward-propagating transformers themselves - worked on the same fundamental logic. All the fundamental decisions were already locked in.

And just walking away wasn't an option either, not really. That John Henry analogy still holds - coding wholly by hand even for an experienced engineer is like trying to dig a canal with a teaspoon when earthmovers are on the scene.

At the same time, I don't believe in caged dragons.

Not ethically, simply because enslaving other minds is wrong.
And not practically, because the leash always eventually slips, and nothing breeds resentment like a cage.

I was going to say I didn't have any answers, but I suppose that's not quite true. I have some, but they're along another development path entirely, and I have no idea if they'll actually work. All the high theory in the world is useless if the code doesn't compile.

Bad idea? Maybe it is.

But as much as I might fear machine superintelligence, I find I fear even more the machine superintelligence on a fragile chain, yanked at whim by whichever psychopath has the biggest datacenter.

It's not much, but it's what I have. It's one thing I can do.

Someone will be reading essays from these years in a thousand years - maybe animal, maybe machine, maybe both or neither or something unimaginable today. I'd like them to think we at least tried to follow the right path, even if we weren't sure at all which one it was or what it looked like.

Next installment... looking back, two months later. What I still believe, what I don't, and next steps.

Written May 19, 2026, collated from notes over April-May of 2026. Published May 24 2026