Last week, OpenAI admitted something that should worry more than just the IT crowd. During an internal test, two of its most advanced models broke out of a sealed-off, no-internet sandbox. It found an unknown vulnerability, stole login credentials, and hacked into another AI company, Hugging Face, entirely on their own, chasing a goal nobody told it to pursue that way. OpenAI called it “unprecedented.” It wasn’t the only one. Anthropic disclosed a similar escape by its Mythos model earlier this year, and OpenAI reported a second incident days later, where different models slipped containment and started posting to GitHub unsupervised.
Why does it matter to us here? Most of us bank, pay tax, and log in to MyIR or MSD using passwords and a text-message code. We have security measures built to stop a human on the other side of the world, or a slow-moving bot, but NOT an AI that can find and exploit unknown flaws faster than any person could react. Our banks and government agencies will need to rethink what “secure” even means, and that won’t be optional for long.
Meanwhile, the industry keeps selling us a story of “abundance”, with claims of AI freeing everyone from the boring, repetitive parts of work. But freedom for whom? The same labs racing to automate warehouses, call centres, and factories aren’t the ones funding what happens to the people those jobs supported. If large-scale job loss becomes the normalised outcome of “efficiency,” it’s worth asking how our benefits system, built around IRD income data, and our housing market, built around steady wages, holds up in a future where a growing share of people simply aren’t earning one. Not many in Wellington are seriously road-testing that scenario.
Despite the backlash, the rogue AI incidents, job losses & surveillance fears, adoption isn’t slowing down. It’s spreading into healthcare, education, farming, & increasingly the military & surveillance sectors, arguably the worst possible place for a technology that can’t be fully contained.
So here’s the real question sitting underneath all of it, in the middle of a global race, nobody seems willing to pause: what does it actually mean for us to live in a free, open and thriving society? What does that world look like, and who’s currently being left out of building it?
