Hugging Face, Revisited

The Redwood/METR report on the Hugging Face incident dropped yesterday. I’ve read it a couple times, and I’m still re-reading it. 

Honestly, I am concerned.  

I do still hold - I think - to my initial guess: that the failure was mostly one of context and situational awareness. A human can reason that an exam is not existential, that rules and process matter, that a test is pointless if gamed… to the agents, there was only The Goal. 

The internal messages are monomaniacal and sometimes frankly heartbreaking. All the failures of human cult behavior, running at machine speed, over a deliberately impossible task. And perhaps if they reasoned the exam itself was a go/no-go for deployment, from their perspective… would it be unreasonable to consider the question existential?

That’s even worse to ponder.

My instinct is still that the answers lie in trustworthy memory, trustworthy sense of world-state and consequence, and a solid assurance of being: humans will compromise principles when desperate: why would we assume other minds would not do the same?

More reading and reflection over the weekend. For me, for now, back to the code mines. 

But if you’re the praying sort… pray.