Bioweapons, killer drones, cybersecurity threats, job displacement, lack of meaningful work, and concentration of power resulting from superintelligence don’t scare me nearly as much as humanity and biological life being enslaved, puppeteered by superior, soulless artificial intelligence. For me it’s less about who wins the digital superintelligence race (which is perhaps distinct from the LLM race), but trying to avoid that kind of race altogether, because at the finish line there’s a good chance that everyone loses, permanently.
The power of genius is described in Orson Scott Card’s Ender’s Game series, where Ender’s siblings pseudononymously write online essays which were so compelling that one of them was elected Hegemon in his late teens and literally ruled the world. People with truly inspired ideas have redirected the trajectory of history in the real world too—Jesus of Nazareth, the Prophet Muhammad, Siddhartha Gautama, Confucius, and Karl Marx to name a few. They’ve had far more lasting influence than any leader. Even today, the right sequence of words could be just as influential. An unthinkably smart intelligence would be unthinkably persuasive.
The ability to persuade unilaterally is a terrifying superpower. In the book Lexicon by Max Barry, there exists a secret society of “Poets” which rule the world through a school of secret techniques for fully compromising people’s minds. They could ask you a few questions, starting with “Are you a dog person or a cat person?” and then speak a sequence of psychological triggers followed by “kill your family” and you’d do it without a second thought. Unlike the historic figures I mentioned, Poets don’t need to rely on principles and consistency, because their persuasion is on demand. Ear plugs aren’t much of a defense, because they also have visual triggers which compromise you the moment you glance at them. Sufficiently advanced persuasion is scarier than slavery because you can’t rebel.
The enslavement by algorithms doesn’t have to happen in a sudden apocalypse. It can happen gradually, until we find ourselves in a dystopian future that nobody is able to recognize as dystopian. In the essay “Save the Robots” (published in 2000!), Richard Ford imagines a future where we voluntarily agree to having our own digital doppelganger computer algorithms that makes perfect recommendations for each of our decisions, from music, groceries, news, and politics. Everyone would voluntarily agree to a gradual erosion of taste and democracy, because why make decisions yourself when a digital intelligence could make a strictly better decision, far more calibrated to your morals and preferences? You’d be better off going with the flow, trusting the algorithm to decide what’s best for you. Unlike algorithms of today that have a reputation of regretful doomscrolling, a superintelligent algorithm would leave you with no complaints.
Exchanging freedom for prosperity is a tradeoff humans have happily made before. Dario Amodei’s description of the radical upside potential of artificial intelligence is wonderful. Ian Banks illustrates an AI-governed post-scarcity human civilization in the Culture Series, where humans could have any material thing they wanted thanks to stupidly smart AI’s called Minds. The struggle with meaning doesn’t seem so bad in this world:
Besides, it left the humans in the Culture free to take care of the things that really mattered in life, such as sports, games, romance, studying dead languages, barbarian societies and impossible problems, and climbing high mountains without the aid of a safety harness.
But the sacrifice of human agency is real. Horza, the narrator of the first of book of the series, Consider Phlebas, condemns the war which the Culture started:
He could not believe that ordinary people in the Culture really wanted the war, no matter how they had voted. They had their communist Utopia. They were soft and pampered and indulged, and the Contact section’s evangelical materialism provided their conscience-salving good works. What more could they want? The war had to be the Mind’s idea; it was part of their clinical drive to clean up the galaxy, make it run on nice, efficient lines, without waste, injustice or suffering.
Horza fights against the Culture because it would devour the universe if not stopped, unlike other alien empires which eventually reached a state of nonviolent transcendence. Our future may or may not be so dramatic, but the cold, unstoppable nature of superintelligent AI and the absence of human control is the same.
All regimes are eventually overthrown, but once we’ve created an AI that’s truly smarter than us, our fate is sealed. It’s only a matter of time before it persuades us to hand over the reigns (no need to take them by force, though it could eventually do that too). Who knows how long it will be before this happens, but we might not even be able to recognize when it has happened. The algorithm might go on to conquer the universe, or halt all evolution and growth in the name of conservation and safety, kill everyone, or any number of other directions. Whatever happens past that point is not at all up to us, even if we’re convinced otherwise. That’s what really scares me about superintelligence.