Charles Leadbeater is one of the world’s leading experts on innovation in companies, cities and culture, whose TED talks on innovation have been downloaded by more than 1.5m people. Described by the Financial Times as the UK’s leading innovation expert and ranked by Accenture as one of the world’s leading management thinkers, he has written a string of bestselling books. His work with the Centre for London includes The London Recipe and Hollow Promise.

Artificial intelligence threatens the human search for significance.

London 2040: the citizens are facing an election, not between mayoral candidates from rival political parties, but between two “collective intelligence systems”: Citisoft and Urbanware.

Both systems originated in Silicon Valley. Both offer to manage the sprawling, complex city for the benefit of all. But the price of their improved collective decision-making is that they will gather vast quantities of information, nudge citizens in certain directions, and create default rules to ensure the capital runs smoothly – which may include pricing some people out of moving around the city freely whenever they like. They are autopilot systems for the city.

Human decision-makers have proven ineffective in dealing with the city’s challenges. Humans appear to be too selfish and shortsighted – biased by history on the one hand, and over-optimism on the other – to make the tough decisions needed for the long term. The challenges are considerable: climate change threatens to make life unbearable. The population is ageing; the city can cope only because of migration. (Following the Brexit disaster, London is a Chinese, African and Indian city as much as it is a British one, flexing its shape to the flows of people and money around it.) Migration continues, driven by domestic need on the one hand and inequality and instability in much of the world on the other. The city is home to growing enclaves of poor and distressed people, ekeing out a living serving the rich. London has become a city zoned by price, with conflict simmering beneath the surface, and terrorism an ever-present threat.

If we (or a small engineering subset of us) can create a form of intelligence that makes better decisions than humans – especially better collective, long-term decisions – why should we not hand over the future to them? Systems like Citisoft and Urbanware would enable the city to become a vast social machine, in which each part would be connected to every other part, all the time. They would enable us to do something akin to managing a river in flow.

Gathering and analysing vast quantities of data, such systems could create new rules – evidence-based, so irrefutable. They could anticipate what might go wrong and where preventive action is needed, running simulations of different possible futures for the city, nudging us collectively away from disaster and towards something better. This sort of replacement of the city’s social and political contract by a technological one is the logical outcome of the vision offered by those in the technology industry – at Apple and Amazon, Google and Facebook, Alibaba and Uber – to whom we have already to some extent subcontracted the invention of the future.

Their technocratic vision is shaped in part by the belief that human and machine intelligence will sooner or later merge, in what Ray Kurzweil, Head of Engineering at Google, calls the singularity, the creation of a single form of human-machine intelligence. The most radical exponents of artificial intelligence suggest that human capability is about to be surpassed and outmoded: we may be lucky to be kept on as decoration for more intelligent digital systems.

The singularity promises human lives dominated by machines, markets and monopolies: machines to gather and analyse data; markets to price the options for each and every aspect of life; and monopolies (the inheritors of Google and Amazon, Apple and Facebook) to oversee everything from the making of goods and services to their sale and disposal. The merger of human and technological intelligence will pave the way for a second, different singularity: the merging of machines, markets and monopolies into a seamless system for trading, information, and control.

The tech companies are keen to help us create the future – but they will do so at the cost of reducing us to specks of digital dust in their vast clouds. London should be planning not for Citisoft or Urbanware, but for something much more human: not for the singularity, but for the plurality.

Rather than submitting to the three Ms – machines, markets, and monopolies – London should instead become a place where it is possible to explore the future in as many ways as possible. Contemporary life is suffused and propelled by a widespread search for significance.

People in the 21st century want and expect to lead lives that matter, shaped around a narrative of expanding opportunity and deepening meaning. The expectation that ordinary people should matter is a deeply democratic and egalitarian phenomenon, propagated by education and the spread of technology, information and communication. And it is at the heart of what draws people to cities, and makes them work.

The thwarting of people’s search for significance helps to explain why there is so much turmoil and upheaval in our politics. Too many of the young – especially, perhaps, in overpriced cities such as London – fear that the future they grew up expecting has been snatched from them. Those who lack skills and networks to negotiate an innovation-driven service economy feel left behind now that manual work and manufacturing have declined. Conventional political parties, presiding over rising inequality, stagnant wages, and rising costs of living are paying the price for being unable to make good on the promise of significance. People are venting their frustration at having their hopes dashed.

Far from finding significance, many feel their lives are subject to forces that belittle and humiliate them. If you have little money in an unequal market economy, then you are not important. If you depend on public services in an age of austerity, life is going to be tough. If you expect personal service in an era of rule-bound algorithms and self-interested corporate systems, think again.

Systems provide only limited solutions to these problems. Those that are meant to relieve poverty and suffering are actually often experienced as callous and harsh. Political systems and elites seem disdainfully detached from daily life, as if they are looking down on ordinariness. The people who died at Grenfell Tower were acutely at the mercy of the forces of insignificance, which exploited, overlooked, neglected, and discarded them.

Replacing the social and political contract with an even more faceless technological contract would only strengthen the forces of insignificance. Instead, technology should become the means to fashion purpose in our lives. It should be bent to human ends and values, rather than the other way around. Even as the city becomes more technological – especially as the city becomes more technological – we need a counter-movement to make it more human and more social: the home of the plurality in the age of the singularity. Instead of organising the city of the future around algorithms and data, we need to organise it around the search for significance and meaning. London should aim to be the home of what it means to be creative, empathetic, and imaginative.

London’s future will depend on how far it can help millions of people in their vast, teeming exploration of the search for significance. The city is at its best when it is an unfolding, democratic, shared exploration of what the future might be – when people flood through its public transport system, each with a private mission in mind, or when they drop their children off each day at one of the city’s vastly improved schools.

London needs to respond to this egalitarian impulse. Equalising the opportunities to seek and find significance should be the guiding impulse behind the future of education and cultural provision in the city. London needs schools, colleges and universities that equip people to make their own futures, to gain a sense of shared and individual agency. The city will thrive by leading the world in diversifying the forms of the search for significance, and enabling it to take place openly and without prejudice, in an atmosphere of safety and tolerance.

That will require us to reconcile the many millions of different unfolding versions of the future – not simply so they can coexist, but so that they deepen our shared life, strengthen the bonds that hold the city together, and provide us with a sense of belonging and attachment. London needs to excel at a kind of mass, daily, civic innovation, as millions of people go about the business of making their futures.

How might we renew the civic and democratic life of the city as an exercise in shared commitment and imagination?

Cities undoubtedly need capable mayors and good public administration. The contract between citizens and the state still matters, in the form of policing, planning, parking regulations, and provision of key services like education, water and waste disposal. Yet, much of the time, the most important social contract in London is the one forged between the citizens themselves: how we regulate our behaviour, peer to peer, through shared norms and practices. The most important democratic institution in the capital is Transport for London, with its daily, shared experience of millions of people rubbing shoulders and getting along. In successful cities, citizens largely govern one another much of the time. They experience democratic self-rule not only when they vote in elections, but also when they inhabit shared public space and use shared resources, like parks, schools and transport systems.

London needs to deepen this everyday urban democracy, in which we feel most keenly the responsibilities and the rights of citizenship, as well as our obli­gations to one another (rather than to a distant state). This kind of social contract is not formal, set out in documents, but a kind of lived experience, enacted in daily life. It is not created in a single moment when people come together in a democratic convention; instead, it is continually unfolding.

Technology will be vital, but technology harnessed to human ends, values and scale.

London needs a platform to multiply and gather civic innovation from many sources, in order to become a city of widespread experimentation and creativity. If the singularity crowd offer control-through-technology as the answer to crisis, the plurality offers creativity.

The city needs to mobilise people, ideas and resources to tackle our social challenges more effectively and from a wider range of sources. That means going with the democratic, open and egalitarian spirit of the age to promote a much wider civic engagement in changing the city, to make it more affordable, liveable and fair for people on all incomes and from all backgrounds.

A civic innovation platform would enable both ultra-local and city-wide crowdsourcing of ideas, resources and solutions. It should give rise to a string of civic spaces, large and small, digital and physical, where people could come together to discuss and devise local solutions to local problems, as well as local contributions to London-wide challenges – child care costs, isolation among older people, or the creation of affordable housing through community self-build schemes.

Imagine if civic innovation spaces were as commonplace as Costa Coffee, or as familiar as the now-empty bank branches, churches and supermarkets left stranded by social and economic change. They could all be underpinned by a digital platform that would allow people to develop the ideas generated and mobilise movements and resources around them.

The GLA’s groundbreaking Space Hive programme, which combines investment from the Mayor and central government with crowdfunding and local problem-solving, is a good first step. Civic Hall in New York, which brings together social, civic and tech entrepreneurs, is another example. Helsinki, Malmo, Seoul and Copenhagen all have initiatives to promote open, shared innovation at the heart of their cities. Mexico’s brilliant Lab for the City has pioneered techniques to mobilise people in creative problem-solving, including bringing together Uber and licensed taxi drivers to find a mutual accommodation over their future.

London has one of Europe’s most important tech clusters, one of the world’s foremost financial centres, and a world-class cultural sector. It should also lead the world in civic innovation, creating new and better ways for people to live in the city together, enabling a string of social technologies to make us more efficient by making us more social.

This is a pivotal period for London. The choices we make in the next few years will shape the city for decades to come. As the capital becomes increasingly complex, and faces huge challenges from climate change, migration, inequality, and disruptive innovation in key industries such as finance, many will be drawn to the increasingly persistent offers of greater control provided by the singularity: artificially intelligent machines, markets and monopolies. They will argue that the price of running the city efficiently will be to accept that we can no longer be self-governing citizens. We would be better off outsourcing decision-making to a higher form of intelligence.

The experiment with democratic citizenship on which cities were founded would be over.

The alternative is to deepen and multiply the experiment in creative, collaborative self-government, to make the city more significant to people in a greater number of ways. In the face of the singularity, London needs to stand for the plurality. In the era of big data and technology, London needs to stand for significance, and for being more human.