So it turns out Roombas take photographs that they’re not supposed to be taking and that the photographs end up available to outside viewers, despite all sorts of assurances that nothing of the sort would happen. Well, not quite, maybe, considering that the company in this case insists that all the photographs were authorized and the training was permitted. But what can happen in a planned training can happen generally.
Whatever the device—Alexa, Siri, Roomba, Ring, you name it, if it’s got a microphone or it’s got a camera and it’s got connectivity of any kind, I think it’s safe to say now that it has at some point been doing things that the owners don’t want the device to be doing in terms of surveillance of the household where the device is working. You can forget privacy policies and any other terms of service: somewhere down there in the fine print, the company that sold you the device reserves the right to take your device’s data and use it to further refine their own technology—and to sell that data to other companies that will use it to train other devices, other software. Since nobody really has eyes on the workers who are the primary recipients of that incoming data, a fair amount of what gets pulled in gets pulled out by prurient individuals and stored somewhere that it’s not supposed to be stored. There are a lot of stories out there over the last two decades about that kind of behavior.
In the particular case at hand, iRobot insists that the controversial photos are not from its general consumer products but special training devices where the users were completely aware that the device was transmitting images, so no violation of policy and no unwanted or unsanctioned intrusion on privacy.
However, I don’t think the main issue that worries me is actually the violation of privacy. For all sorts of reasons, lots of people seem relatively indifferent to being under constant surveillance by devices and technologies that they’ve voluntarily brought into their homes. Nobody needs Alexa or Google Home (and apparently sales on ‘smart speakers’ are slumping as the novelty wears off and people realize that they’re just super-elaborate versions of The Clapper) but the people who bought it mostly didn’t care it was listening to them even when they didn’t mean for it to be listening. I think it’s a combination of people thinking that what they say and do day in and day out can’t possibly be interesting to a corporate eavesdropper and feeling that if someone wants to listen to them or spy on them, they’re going to have a hundred other ways to do it in our present technosocial infrastructure. Those are both reasonable enough thoughts, but there are terrible dangers lurking inside ubiquitous surveillance.
That’s where my real worry comes in. It’s not the erosion of 20th Century privacy that’s the issue, it’s what companies are actually doing with all that data. Sure, some of what they’re doing is refining the functioning of their own product, so that Roomba maybe learns not to plow through a dog turd and what to do when the toddler drops a jacket along the usual cleaning route. But the current story-cycle about the photographs that have been stored makes clear that the data that these devices collect, along with the vast reams of data being scraped everyday from the Internet, is being used to train many kinds of specialist AIs with machine learning.
That’s what really worries me, what the “safe AI” movement is also taking note of. AI is being trained all over the world. We can see what’s going on in many academic institutions and even with some of Big Tech but there’s a whole host of smaller third-party companies and groups that are doing whatever the hell they want. There’s no meaningful regulation, there’s no strong ethical guidelines.
Our imagination about what our data is used for is still trapped in the 20th Century—we think it’s about selling us products or spying on us to predict our behavior. That’s not it any longer. It’s about building AIs that might well be used to lower our salaries or fire us, about AIs that will be a black box even to the people who’ve made them, about a world full of attractors that we’ll be pulled towards that no one in particular intended or wanted. The stock market blips that were attributed to early generations of high-frequency trading programs are only a small foretaste of where we might find ourselves in the very near future.
In a world whose scope is so vast and thus unmanageable by our ordinary intuitions, we may end up resigned to accepting strange and unintended emergent behavior by autonomous devices that arises out of unaccountable processes of training their systems with data that they shouldn’t have been vouchsafed in the first place. I worry less about a next-generation robot vacuum taking a picture of me and more about a next-generation robot vacuum interacting with the medical testing unit in my toilet and deciding that it would be best if the cat died, thereby reducing my histamine levels and getting rid of the cat hair that slows the vacuum’s efficiency. That won’t be a decision in the sense of a sapient general-purpose AI mimicking our own reasoning—it’ll be a weird shortcut that improves the machine-learnt algorithm’s efficiency. We’re not so much on the cusp of The Singularity as we are on the edge of a kind of alchemy, not one gigantic leap into an utterly different future, but instead a million unpredictable emergences born out of the careless, ceaseless, heedless motion of data capitalism creating attractors it doesn’t understand and can’t predict.
Image credit: Photo by Denny Müller on Unsplash