The AI future some researchers worry about

Artificial intelligence is already reshaping the economy, and it’s not going anywhere. But what it becomes in the years ahead is far less certain. As tech companies race to build more powerful systems, and as debates grow over everything from autonomous weapons to government surveillance, some researchers are warning that the risks may extend far beyond job loss or misinformation.

Experts at the Machine Intelligence Research Institute take the potential threat of an AI-caused catastrophe seriously. For decades, MIRI worked to identify and manage potential risks — all with an eye on preventing future AI-driven human extinction.

“Marketplace Morning Report” host David Brancaccio spoke with MIRI president Nate Soares. The following is an edited transcript of their conversation.

David Brancaccio: So, I understand this. Does the Machine Intelligence Research Institute right now want our use of AI to stop, right now?

Nate Soares: It's not that the current use needs to stop. It's just that the race to make smarter and smarter machines that nobody understands need to stop. You don't need to give up ChatGPT. You don't need to give up self-driving cars. You don't need to give up on the medical advancements. It's really just this race to assemble far more specialized computer chips than anyone has assembled before, and do a massive training run where no one knows the consequences. It's only that part on the frontier where we need to be pausing,

Brancaccio: Who would be at the table, if you convened people who would chew on this issue?

Soares: I mean, it's not just computer scientists — as much as we love them. I mean, it has to be politicians, has to be world leaders, and, in fact, it has to be international. You need international collaboration here. We've seen people start trying to shut down data centers in their towns. And unfortunately, an AI does not need to be running in the data center in your town in order to take your job. If these AIs are running anywhere on the planet, they can have the effects that people are hoping they won't have, and frankly, we need global coordination.

Brancaccio: So I'm clear, when you talk of the potential for an existential risk here, that's not hyperbole?

Soares: That's correct. We're talking about the type of intelligence that humans have and that mice lack. That is something that these researchers are desperately trying to get into machines, and if they do, we're looking at machines that can invent their own technology, that can invent their own infrastructure. The issue then is not that they would hate us. It's just that we die like any other animal, that where we took their habitat did.

Brancaccio: I heard a retired senior U.S. diplomat just the other day refer to the Eisenhower administration in 1957, knowing that atomic weapons could kill us all, embrace the International Atomic Energy Agency. That diplomat said, “We need that for AI.” Would that be a model?

Soares: I think that would be a great model. You know, AI computer chips these days are very highly specialized. They can be made in only a few places in the world, using tools that only exist in sometimes one place in the world, and to train a new AI, you don't just need a center for use. You need an enormous data center that takes as much electricity as a city, it would be pretty easy to control these computer chips and prevent people from these enormous new training runs where no one knows what's going to happen. All we really need is the political will.