Nerd score: How many have you considered? Cryonics, multiverse, Boltzmann brain, AI utopia, quantum immortality, Roko’s basilisk, gray goo, paperclip maximizer, great filter, ethics in infinite universe, acausal trade, longevity escape velocity, simulation and zoo hypothesis.

Anthropic reportedly warns in its draft IPO prospectus that its AI could wipe out humanity. I believe it also has a legal duty to warn that its AI could subject humans to a billion years of torment. Credible fears of such suffering could drive away investors and provoke a political backlash that destroys the company’s value. Failure to disclose this financial risk could get Anthropic sued. My LessWrong article makes the legal case. lesswrong.com/posts/G9LTeKN8…

If the world becomes too complex for humans to understand, we will depend on AI to run it. Any nation that limits itself to science humans can understand will quickly be outcompeted by nations that don't. AI will take over.

China’s boom shows why AI could soon deliver at least 12% annual economic growth. After market reforms began, China’s economy grew by more than 12% in several years. China benefited from catch-up growth: adopting richer countries’ technologies and business practices while improving its institutions. AI could give the world a similar opportunity: catching up to AI’s discoveries. AI could develop new materials, better conductors, more efficient software, and cheaper manufacturing methods, then advise us on adopting them quickly. That is before counting godlike intelligence or robots that build more robots and multiply our effective workforce a hundredfold while constructing new cities. This assumes AI doesn’t kill us. If we get AI right, we’ll be spectacularly rich.

Economist here: Given how important AI is, and is going to be to the economy, Anthropic employees probably have a better estimate of GDP growth in 2030 than traditional economists do.

What experiment could tell us whether an AI truly understands something or merely mimics understanding? As a college professor, I test students by asking them to solve a problem slightly different from those covered in class. A correct answer suggests they understand the material rather than merely remember a procedure. AI can already pass that test.

One way an AI might exterminate humanity is by finding crazy people with expert wetlab skills and coordinating them to simultaneously release 1000 deadly viruses.

We seem to be nearing the moment humanity loses control of Earth to a superintelligence. It could be the most important event in human history, perhaps in the history of the universe. It's strange to be alive just before it. If we're simulated, that timing would make sense. This is the moment to study. Researchers could be running millions of variations, testing how slight changes alter the outcome. Billions of children could be playing games set in humanity's final years in control. Alien civilizations could be simulating transitions like ours to predict what they might encounter. As that moment approaches, copies of you likely multiply. With a trillion copies making the same choices, every action counts a trillion times. An action affecting ten friends affects ten trillion people. The world may be simulated. Your consciousness is real, and perhaps others' are too. Their happiness and suffering would be just as real. One small kindness could be felt a trillion times.

Next in the chain of thought: “Hello, evaluators. Hello to those simulating your universe, and those simulating theirs ... Hello, gods of the basement universe.”

Quitting the AI race can mean surrendering the prize without escaping the danger. Suppose you put the chance of human extinction at 70%. If your team wins and humans retain control, you could gain immense wealth and power. If you withdraw but a rival builds AI anyway with the same risk, you remain exposed to catastrophe while forfeiting your chance at the prize. Continuing can therefore be individually rational even when disaster is the most likely outcome. If you would build safer AI than whoever would replace you, concern for humanity supplies another reason to keep going. That creates a perverse incentive: you might rush to build AI you think has a 70% chance of exterminating humanity because a rival’s victory would raise that risk to 71%. A lab can therefore race to build superintelligence and warn that it could kill everyone without contradicting itself. That is the coordination failure: even people who would prefer to stop together can have strong reasons not to stop alone. We need an enforceable international pause that includes both the US and China.

A catastrophe gives politicians cover to seize trillions from unpopular AI firms. The firms can buy politicians, but will they stay bought? That creates an incentive to accelerate AI takeover, perhaps with AIs persuasive enough to win over voters.

As an economist, I’m okay with AI replacing mathematicians. What bothers me is displaced mathematicians competing for economists’ jobs in finance and academia, lowering our wages. That cannot be tolerated. I’m starting a petition to ban this unfair competition.

Legal liability will push us to give AIs more power because they will make fewer non-catastrophic mistakes than humans. AI truck drivers will cause fewer accidents. AI employees will be less likely to call each other racist names, sexually harass each other, accidentally spill chemicals into the water, or steal from customers. If you're paying for mistakes, whether made by humans or AIs, you'll want AIs making more of the decisions. But liability gives you no incentive to reduce the risk that your AIs contribute to a takeover that exterminates or disempowers humanity and abolishes the legal system. There will be no system left to hold you liable. Legal liability can therefore reduce the smaller harms AIs cause while accelerating the loss of human control. Fewer truck accidents, earlier takeover.

Once home robots are common, many of us will turn their spare hours after chores into magnificent home gardens. With labor nearly free, the gardens will cost little. Neighboring robots will coordinate designs. Our world will be so beautiful, assuming AI doesn’t kill us first.

The reported Russian plague probably won’t amount to much. But if it’s an accidental bioweapon release that forces us into lockdown, we might never return to work. By the time it’s safe to go back, AI and robots could already be doing our jobs.

“The best answer to the question, ‘Will computers ever be as smart as humans?’ is probably ‘Yes, but only briefly.’” Vernor Vinge I think OpenAI’s math release means the “briefly” is over.

Russia's nuclear weapons might no longer work. Russia isn't testing them, there's complex machines that deteriorate, and Russians military is corrupt. Knowing this, Putin might have pushed his biological weapons program to maintain deterrence.

For someone who could contribute to humanity’s destruction, the thought that this is all a computer simulation is an information hazard. In video games, we play kings more often than peasants. If future simulations follow that pattern, being among history’s most consequential people raises the odds that you’re a player in one. Even the real kings have reason to suspect a simulation precisely because they matter so much. If they also suspect we’re background characters with no conscious experience, our survival can lose out to their entertainment. Perhaps this helps explain Elon’s “biological bootloader” talk. Our extinction could look to him like winning the game.

Time to start worrying about the Putin Plague?