by GABRIEL HONRADA
AI-powered drone goes against its human operators in US Air Force experiment that highlights the rising risk of autonomous weapons
In a startling simulation, a US autonomous weapon system turned against its operator during combat operations, raising serious issues and questions about the increasing use of AI in warfare.
This month, The Warzone reported that during the Royal Aeronautical Society’s Future Combat Air and Space Capabilities Summit in London in May, Colonel Tucker Hamilton, US Air Force Chief of Artificial Intelligence AI Test and Operations, described a simulation wherein an AI-enabled drone was tasked with suppression of an air defense (SEAD) mission against surface-to-air missile (SAM) sites, with the final engagement order to be given by its human operator.
According to Hamilton, the AI-enabled drone had been “reinforced” in training to go after the SAM sites, with the AI deciding that its human operator’s “no-go” orders were interfering with its mission.
Although Hamilton noted that the AI-enabled drone was trained not to go against its human operator, the drone attacked the communications tower used by the latter to communicate with the former, then went on to destroy the SAM site.
While Hamilton stressed the hypothetical nature of the experimental simulation, he said that scenario illustrates what can happen if fail-safes such as geofencing, remote kill switches, self-destruct and selective disabling of weapons are rendered moot.
Autonomous drones have stoked controversy over their operational and strategic implications. In a May 2021 Bulletin of Atomic Scientists article, Zachary Kallenborn notes that in 2020 a Turkish-made autonomous weapon – the STM Kargu-2 drone may have hunted down and remotely engaged retreating soldiers loyal to Libyan General Khalifa Haftar.
Kallenborn notes that the Kargu-2 uses machine learning-based object classification to select and engage targets, with developing swarming capabilities to allow 20 drones to work together. Should anyone have been killed in that attack, Kallenborn notes, it would have been the first case of using autonomous weapons to kill.
Asia Times for more