Sitting in a U.N. committee meeting in Geneva earlier this year in a session on Lethal Autonomous Weapons Systems (aka killer robots), I was shocked to hear the American delegates claim that AI-powered automated warfare could be safe and reliable. Not only are they wrong, but their thinking endangers us all. It’s the same logic that led to the Cold War, the nuclear arms race and the Doomsday Clock.
I quit my job at a young, promising tech company in January in protest precisely because I was concerned about how the Pentagon might use AI in warfare and how the business I was part of might contribute to it. I have seen close up the perils of this unreliable but powerful technology, and I have since joined the International Committee for Robot Arms Control (ICRAC) and the Campaign to Stop Killer Robotsto make sure that AI is used responsibly, even in cases of war. It was because of this protest and my objection to autonomous weaponry that I attended the U.N. conference as part of the campaign’s delegation.
The best way to deal with this technology isn’t for the U.S. to propel the killer robot arms race by building bigger, faster, deadlier machines of our own. It’s to cut it short.
With Russia actively seeking to derail international negotiations to regulate Lethal Autonomous Weapons Systems, or LAWS, and China churning out ever-improving AI-powered drones, the fears of these two countries’ potential use of killer robots are understandable. But the best way to deal with it isn’t for the U.S. to propel the killer robot arms race by building bigger, faster, deadlier machines of our own. It’s to cut it short.
We’ve been down the path of escalation before, and we can be smarter this time around. Today, we should put in place international laws under the strict enforcement of the U.N. that ban the development of these systems and condemn their use globally. If our goal is the easing of worldwide tensions, and we want Russia and China to adhere to the ban, we can achieve this by limiting the military advantages of this technology. That will make it less appealing for adversaries to pursue — and less worth the price they’ll pay for being outside the bounds of an international accord.
The technical limitations of LAWS provide a clear opportunity for keeping the lid shut on this Pandora’s box: We should devote our considerable hi-tech R&D minds and resources into developing robust countermeasures rather than offensive ones. If our enemies build Terminators, we can fight like John Connor, who valiantly defended humankind in the rebellion against its would-be Terminator overlords.
What makes me so confident that the John Connor route is viable?
Before I left my job as head of data operations at the AI startup Clarifai, I saw firsthand the many spectacular and unpredictable ways that AI can fail. It was my job to find image data that would fill the gaps to improve performance in the computer vision models we built. For every mistake we could correct by adding more photos to the data set, a new, unexpected issue would arise. It’s a little like cooking without a recipe — if that recipe were brand new, there were millions of ingredients, and someone forgot to label all the jars.
AI models are, by definition, predictive technology. They can only predict objects very similar to the objects they’ve previously seen, and it’s impossible to cover every potential scenario that a model might encounter. The camera angle, light, proximity to objects, camera quality and weather conditions are just some of the factors that affect the accuracy of a computer model trying to identify its target.
This is what researchers mean when they call such models “brittle”: They tend to crack when faced with a scenario slightly different from the conditions introduced during the model’s construction. This is doubly true when talking about warfare, since battlefield conditions are rarely predictable, and the enemy will always try to find a way to exploit this weakness.
In fact, there is an entire field of research dedicated to fooling AI models for computer vision — the very models the Pentagon would need to install in killer robots so they can aim at targets. To date, no computer vision model has proved bulletproof against the exploits researchers call “adversarial attacks,” something akin to hacking to purposely fool the computer vision program.
In World War II, we used chaff (a cloud of thin pieces of aluminum) to prevent enemy radar from targeting our airplanes. That same principle can be applied to adversarial attacks against killer robots. In the analogous case for AI, images with a little “noise” (static or some other pixel-based disturbance) added to the photo have been shown to defeat AI vision models — even if the image looks fine to a human eye. So picture a platoon of tanks riding under a netting of multicolored, reflective confetti or something equally out of place. If the robots haven’t been trained to deal with that, the soldiers underneath will be practically invisible.
Research also suggests that it’s not shape or color but texture that allows AI vision models to lock onto a given object before it launches an attack. If the Terminator is looking for skin, let’s bring fur to the battle. If it’s programmed to expect sand, litter the battlefield with purple AstroTurf.
If we want to get sophisticated, let’s invent a living, mutable camouflage that morphs into generative patterns and textures. Or equip our troops with unexpected objects, like, say, pink inflatable porcupines affixed to armored vehicles. Because those objects don’t resemble anything that was used to train the targeting algorithm, the computer will assume the humans it sees are somehow different from those it was trained to kill.
Lethal Autonomous Weapons Systems are by their very nature unsafe, and if we allow nations — especially undemocratic ones — to make them, we all stand to lose from their mistakes.
I don’t profess to be an expert in defense strategy, and I’m happy to leave the exact details to the John Connors in assorted U.S. military labs. I am, however, someone with access to the many grave ways computer vision models can fail, be it by adversarial attack or even just the implicit biases of those who seek to make them.
The American delegation’s claim about the viability of warfare in which machines can launch attacks without a human ever making the final decision to pull the trigger demonstrated to me the sincere lack of technical expertise needed to have these conversations in earnest. That’s a major part of what scares me and those like me in the Campaign to Stop Killer Robots. Lethal Autonomous Weapons Systems are by their very nature unsafe, and if we allow nations — especially undemocratic ones — to make them, we all stand to lose from their mistakes.