Using white dots stuck on the tarmac, a recent study from Tencent Keen Security Lab in China pushed a Tesla Model S onto the wrong side of the road. What makes the work so interesting is the researchers didn't have to alter the car's code. They just used its own cameras and sensors, which look for lane markings, against it, so they just TRICKED it; they did not HACK into anything per se.
Ariel Herbert-Voss, a third year PhD student at Harvard University, begs to disagree. She studies adversarial machine learning - where an attacker uses external signals to force an AI system into making an incorrect prediction, like choosing the wrong lane. Ms. Herbert-Voss grew up hacking computers and doesn't see much of a distinction, if any at all, between hacking a system and tricking it.
Hackers usually want to make money, she said, or to "cause some general chaos. In most cases, it just involves fooling a system somehow, and usually, they want to take the path of least resistance. So, if you can fool a car by just having a bunch of stickers on the road, I guarantee you hackers are going do that."
But what are the police going to do about it? The United States introduced anti-hacking laws after members of President Ronald Reagan's administration saw the film War Games, in which a computer almost starts World War III. The Computer Fraud and Abuse Act, implemented in the mid-1980s, made it a federal crime to hack into a computer system. But what about tricking an automated system, without bothering to hack it?
Ryan Calo, co-director of the University of Washington's Tech Policy Lab, recently published a paper asking this question: "Is tricking a robot hacking?" Unlike the traditional understanding of hacking - entering a system, stealing information or changing its code - this threat includes prompting an AI system to make what Mr. Calo called "errors of consequence". "You're not doing it by breaking into the system," he said. "You're just understanding how the model works and then influencing it, affecting it, forcing it to do the wrong thing."While the results could be just as serious as traditional hacking, Mr. Calo and his colleagues are concerned this doesn't fit neatly within current US regulation.
Australian federal law is a little more prepared for this grey area, according to Professor Kieran Tranter, who researches law and technology at the Queensland University of Technology. "Our criminal code prohibits not only getting into the code and changing it but also potentially affecting its inputs. So arguably, doing adversarial machine learning ... or just doing things to confuse the robots could still be covered by the Australian laws," he said. While Australia's criminal law may be broad enough to cover these scenarios, the bigger threat, Dr. Tranter said, is the "known unknown". "Often the most interesting and malevolent uses of technology are the ones no-one's ever thought of," he said.
It is the "known unknowns" that raise another sticky question: Who is responsible when a system can be fooled? For now, companies can sometimes be penalized if they fail to secure their systems against malicious hacking. In Mr. Calo's view, the same ought to apply to systems that are too trickable. He also wants the laws to be clarified so that researchers and others are not criminalized for pushing the boundaries and testing whether such systems can be fooled.
Ms. Herbert-Voss said the Tesla study shows that intelligent systems must be built for robustness: against hackers, certainly, but also hazards as mundane as bad weather. What if, instead of white stickers, a scattering of de-icing salt dragged a self-driving car into oncoming traffic? Apart from manipulating an existing system, researchers are also looking at how AI could be trained or manipulated to make mistakes in the future. Machine learning tools often rely on large datasets to teach themselves about the world - to distinguish a curb from a driveway, or from lane markings, for example, they may need to be trained on millions of such images. But this also provides a vulnerability.
Only remember Tay, Microsoft's ill-fated chatbot, who was designed to learn by interacting with humans on Twitter. It wasn't long before she was tweeting "feminism is cancer". "There are opportunities for you to inject malicious behavior into the very training of the algorithm, which then later will perform the way that you, the attacker, wants," Mr. Calo said. No matter how automated and sophisticated a system is, there will always be ways to exploit it. "I think a lot of people fall into the trap where they think that if it's a decision from a machine, it's infallible," Ms. Herbert-Voss said. "But humans are trickable and so are machines."
