The internet of eyes of ears

If the Cambridge Analytica scandal shocked you, then it might be time to avert your eyes. The notorious data breach and subsequent revelations about Facebook’s outsize influence on political dialogue and leaky control of personal data has rippled across the globe, with troubling instances from Myanmar to Brazil and beyond.

But such scandals could be the tip of the iceberg when it comes to the tech industry’s encroachment into politics and society. Silicon Valley is creeping into the civic space, from transport networks to city design to healthcare, while technical innovations are making the internet not just a thing on a screen but literally in the air around us. We have entered a new, hyper-digital era, in which our data is harvested in myriad ways, through such things as visual recognition technology, verbal assistants (Alexa, Amazon Echo et al), ‘Smart Cities’, 5G (Hello, smart cars), and a universal sensor culture in which even your dog can generate data points. Thanks to artificial intelligence, it is all instantly analysable, and offers intimate layers of insight into people’s lives that could easily have political application.

What could possibly go wrong? The scandals, data breaches, and blind spots unearthed with GAFA (Google, Apple, Facebook and Apple) and Uber took place on their core product platforms, let alone these new technologies. Biases are already starting to emerge in the algorithms, designs, and key functions of these ostensibly commercial products.

The fact is, Silicon Valley is now in a position to wield influence and power in ways not seen before. It is at once government vendor, collaborator, sponsor, communicator, advisor, rival, and enemy on any given day. Its platforms are now integral to democracy, elections, and governments, with funds that dwarf those of the state itself: Apple’s cash reserves in 2017 were more than $250 billion; total Federal Reserves in November 2016 were just over $118 billion.

In some respects, Big Tech companies are trans-border nation states that do not require government approval for their activities. New York even offered Amazon $1.7 billion in incentives to land one of its HQ2 headquarters in Queens, and states across the US have competed to offer tax breaks to the giant. The New York Times has revealed the shocking extent of Facebook’s lobbying and political manipulation since its data crises. Meanwhile, gadgets like Amazon Echo and Google Home hub are being given away at knock-down $30 prices, because they aren’t the product – we are. New consumer technologies such as facial recognition, voice activation, and interactive hubs are already becoming the norm.

At J. Walter Thompson (JWT), where I work, we identified the Internet of Eyes and Ears as one of our biggest consumer tech trends for 2018. The trend explores how new consumer technologies listen to us, watch us, and then respond rapidly, at a remarkable rate. Everyday objects are outfitted with smart cameras and the latest in visual recognition technology, then combined with machine learning to analyse images, facial expressions, emotions, and to identify people (and even pets).

Take facial recognition technology, for instance, which is moving beyond the passport gate to a form of financial ID. In China in 2018, customers visiting KFC branches need only smile to pay for their chicken with Alibaba’s (China’s answer to Amazon) Smile-to-Pay app. The 2018 iPhone X debuted Apple’s facial recognition system to unlock it. Google-owned Nest, which produces ‘smart’ home products including thermostats and security systems, has recently come up with the $299 Nest Cam IQ with built-in facial recognition technology to differentiate between family members and strangers.

“The Internet of Eyes enables inanimate objects to see by leveraging computer vision analysis,” says Evan Nisselson, general partner at visual-technology venture fund LDV Capital. “Inanimate objects with cameras enable companies to own the first step in gathering the data for computer vision and artificial-intelligence algorithms to analyse. Analysis may include object recognition, sentiment analysis, gesture recognition, and many more human actions, which will impact all business sectors and humanity.”

Amazon’s new AI-powered photo recognition shopping aid, Amazon Echo Look, for instance, has the power to be a cognitive consumer census on steroids: It will snap photos of customers, crowd-source opinions on their outfits, keep and analyse visual content, and make tailored recommendations. LDV Capital predicts at least 220 per cent more embedded cameras in the next five years.

Meanwhile, developments in speech recognition and natural language processing (NLP) now allow people to talk to computers and devices in a way that might recently have seemed like a sci-fi movie. This technology helps people to activate products with their voice, converse with virtual butlers, and ask for information they might otherwise have typed into a Google search. But it makes the innocuous-looking products that use it, such as Amazon Echo and apps such as Google Assistant and Apple’s Siri, the tech firms’ ears to our homes: Half of mobile internet searches are now made verbally, according to industry analyst Gartner; 22 million Amazon Echos were sold in 2017 alone.

Pretty soon, Big Tech companies will be able to analyse our reading habits, conversations, and even our political discussions – a new layer in what were previously screen-led interactions.

Could the information gathered by these applications eventually go beyond consumerism and encroach upon our civic engagement – even to shape an electoral campaign? For instance, could dinner-table political conversations be used to create even more hyper-tailored and targeted advertising and messaging? Could photo recognition on connected TVs read emotions during political advertising? It’s not that much of a step – assuming it’s not happening already.

The combining of swathes of personal data with artificial intelligence already enables the creation of millions of personalised, targeted experiences. So designing a unique political candidate who suited the most people in their individual digital setting is not much of a leap. It’s interesting to consider this leaping off the page, too, as future Facebook technology moves into 3-D avatars, augmented reality, virtual reality, and beyond. If Hillary Clinton was criticised for not appearing in key states, a tech-backed candidate could campaign, virtually, everywhere.

When almost one third of the world’s population is on Facebook, some would say Facebook and Amazon are near nation-states already. But their platforms and services are still participatory, limited to certain behaviours such as socialising and shopping. They are, to an extent, monitored by at least a veneer of consumer power, while the biases and beliefs of their founders are confined to their philanthropic and investment ventures. But when these are writ large on society, and start to replace the state, their power will be magnified.

Everything starts innocently, and positively. Why wouldn’t you want data to be freely shared? Why wouldn’t you want seamless service? Many government systems will be made more efficient, automated, and digitised. In the near future, the lines between citizenship, employment, and consumerism could blur into a continuous ecosystem that combines consumer voting, personal government documents all stored on Facebook, and people’s CVs available for public scrutiny.

As consumer tech companies move into financial services and identity verification, higher-income consumers could be treated more favourably. Financial clout and social influence could also be combined and assessed to inform policies or even election campaigning. Government itself could become the ultimate algorithmically driven, constantly-updating consumer brand, affected by consumer sentiment continuously. Forget the referendum – blockchain could power our voting on a multitude of issues instantaneously.

The work of Robert Moses, the legendary mid-20th-century American urban developer, is in many ways a prophetic warning. Moses imposed his vision on post-war America, stripping cities of ‘undesirable’ neighbourhoods, building new modern developments, parks, and swimming pools – and making way for the automobile rather than public transport, which he saw as central to America’s economic future. He cut superhighways over and through existing historic cities. He built new housing projects, social experiments to accommodate the poor, under the banner of ‘urban renewal’, many of which alienated and damaged those same communities. Moses was immune to the existing and largely successful complex ecosystems within cities, which were observed, championed, and highlighted in Jane Jacobs’s famous 1961 book The Death and Life of Great American Cities. Today, Silicon Valley is selling us an updated, data-driven, tech-enabled future with a similar Utopian vision to Moses – with similar prejudices and blind spots. And it’s getting increasingly close to realising these ambitions. Do we really want a sky full of drones delivering everything for us? Should public transport be on-demand? Is the driverless car really the future? Efficiencies, increased sustainability, and technological advances are vital, but as more of life becomes connected, more of our interactions are commercialised, too.

The power dynamic between government and Silicon Valley companies, particularly the bigger brands, continues to shift gears. The large tech companies, compared to other corporations, have shown a greater interest in making political statements, a necessary outgrowth of their self representation as forces for good, not just for profit.

The relationship grows more complex when firms decide to mobilise their audiences when something goes against their interests: “You don’t want this! Write to your governor!” Uber used this tactic to help overturn restrictions on its activities. Airbnb has created an entire platform out of mobilising its users to lobby governments on its behalf about renting out properties. Describing Airbnb-ers as a new sharing economy “guild”, it has made a comprehensive strategy out of rallying consumers to campaign for short-term rentals. Silicon Valley wields more financial clout than many governments; it leads innovation in key sectors that these governments used to; and it is increasingly taking on key tasks of governance. More government contracts, from NASA to healthcare, are going to Silicon Valley because its companies have more money and are leading more innovation. This has prompted federal programmes to actively court and acquire stakes in new tech relevant for military and state use.

The vectors of problem-solving and innovation have flipped. In the UK, the government is using Google’s DeepMind machine-learning for the NHS, and big data software firm Palantir for analytics (the city of New Orleans is reportedly experimenting with Palantir’s predictive policing techniques). Silicon Valley is becoming the Expert Friend – and that power shift is very visible to all of us as citizens. Where once government took us to space, our government scientists built the internet, and our prime ministers strategised, private companies are leading us into the future. It may not be the one we expect.

Adapted excerpt from ‘Silicon States, The Power and Politics of Big Tech and What it Means For our Future’, © 2018 by Lucie Greene. Reprinted by permission of Counterpoint Press