Talk to any cybersecurity professional these days and you’ll be met with exasperation, weariness, and an absence of their typical whimsy. These people are tired. They’re calling the past month “Black October” after AI-detected vulnerabilities emerged with unprecedented rapidity, faster than many could patch. They’ve been driven to discouragement.
This is sort of what happens when you create technology far more powerful than what came before it, and let it loose on a society whose defenses were built for a prior era. Safeguards that seemed solid before AI are now seemingly at risk. To many in the field, a major AI-driven catastrophe seems like a matter of when, not if.
OpenAI’s Hugging Face hack drew the world’s attention because it played into science fiction themes of a motivated artificial intelligence going rogue and committing crimes, but a more likely disaster scenario involves humans using AI to cause chaos, or worse. Instead of misaligned AI turning us to paperclips, the pressing danger is a catastrophic hack performed by humans with AI’s aid, the creation of a novel pathogen with AI’s guidance, a military starting a hot war based on an AI-manipulated signal, or something of that nature.
As such a disaster looms, the AI industry is waiting for it to happen before taking action to prevent it. It’s backwards. Despite pledges to ‘pace the frontier’ the frontier remains largely unpaced. There have been some short delays, yes, but the technology is speeding ahead, with researchers using their latest advances to make the next leap even faster. Everyone is racing forward, with no apparent willingness to coordinate a safer collective strategy, even as the consequences of the race seem clear.
Take little solace from the lack of a catastrophe happening yet. Already, signs of a major impending incident are mounting. Cybersecurity is by far the most concerning area after a proliferation of AI coding tools have landed in hackers’ hands.
A 26-year-old, for instance, allegedly hacked multiple banks in South Korea earlier this year with a Chinese-developed open-source AI tool and Claude Code. Another hacking group used models from Anthropic, SpaceX, and Moonshot AI to attack more than 400 financial targets in Latin America, obtaining passwords, API keys, and backdoors. Another used Anthropic’s technology to target government institutions, chemical manufacturers, and government agencies before being caught.
So far, the hacks haven’t brought down a cornerstone institution. But with coding capabilities advancing at a furious rate, and diffusing via open source, it doesn’t seem wise to bet against it.
The risks of an AI-sparked war is also troubling. A widely-ignored report last month found that armed U.S. troops nearly boarded a Chinese ship, with military planes in the air, only for the operation to be called off at the last second after the military realized it was acting on a chatbot’s inaccurate output. The incident, per the report, “almost started a war.”
Beyond sloppiness, bad actors with the technology in their hands could also spoof attacks or trigger defenses that could cause nation states to exchange fire or worse. Adding advanced AI to a planet where the world’s superpowers have nuclear missiles pointed at each other doesn’t seem wise.
Manufacturing a bioweapon with the assistance of AI is difficult, but perhaps not impossible. To do so, you would need a lab and at least some expertise in the discipline. But add those ingredients, and the possibility for a disaster comes into focus. A scientist, per a recent Anthropic report, used Claude to plan research that could insert mutations into bird flu that could make it more transmissible.
Much of what we know about these risks and attacks comes from the AI labs themselves, which have been commendably transparent on these issues. But when it comes to any hint of coordinated action to counter “the big one” before it happens, the industry has shuffled its feet.
To prevent future AI-driven catastrophes, it seems the world will need to experience one to wake up to the risks. But to see the danger coming just to watch it happen is something we’ll likely look back on with bafflement.
What can AI really do in cybersecurity, and where does the hype fall apart? (sponsor)
AI is changing cybersecurity, but which capabilities live up to the hype? Horizon3 CEO and Co-Founder Snehal Antani joins Alex Kantrowitz to separate frontier model capabilities from the claims. Hear why AI agents fall for security decoys more often than human hackers, how AI accelerates exploit development, and why agents should be treated as insider threats. Get a builder’s perspective on what these developments mean for attackers and defenders alike.
This Week On Big Technology Podcast: This AI Expert Argues We’re Underestimating AI’s Productivity Impact — With Ethan Mollick
Ethan Mollick is a professor at the Wharton School and author of Coexistence: The Next Phase of AI. Mollick joins Big Technology to discuss why AI’s remarkable advances haven’t yet translated into massive productivity gains across the economy. Tune in to hear why companies need to fundamentally rethink how they operate to unlock AI’s full potential, where businesses are already seeing returns, and what organizational transformation could look like. We also cover whether AI development will hit a wall, the rise of personal AI agents, the future of jobs, robotics, education, and the risks of giving AI access to our private information. Hit play for a spirited conversation about what it will take for AI to truly transform the economy and our everyday lives.
You can listen on Apple Podcasts, Spotify, or your podcast app of choice
Thanks again for reading. Please share Big Technology if you like it!
My book Always Day One digs into the tech giants’ inner workings, focusing on automation and culture. I’d be thrilled if you’d give it a read. You can find it here.
Join Big Technology’s Private Discord Server!
Where we’ll talk about this story, the latest in AI, the week’s podcast, and plenty more. You can sign up via the link below: