Is AI going to kill us all?

If you read nothing else

The short version

Part one

What the machine is

A chatbot such as ChatGPT, Claude or Gemini runs on a large language model. Its core job is easy to state. Given some text, guess what comes next.

That is all the toy below does, and it is most of what the real thing does too. The real thing is just far better at guessing.

Nobody wrote the rules

Here is the odd part. No engineer typed in grammar, or facts about cats. They built a vast grid of adjustable numbers, called parameters (175 billion of them in GPT-3, back in 2020, and more since), and had it read a large share of the public internet. Each time it guessed the next word wrong, the numbers were nudged a little. Repeat that trillions of times and the guesses get very good.

To guess the next line of a chemistry answer well, a model has to pick up something that works like a grasp of chemistry. Whether to call that "understanding" is one of the rows you will meet below.

Three facts the whole row rests on

These systems are grown, not built. Nobody designed their behaviour line by line. It emerged from training, and it sometimes surprises the people who did the training.

Nobody can fully read them. Every number inside the model is there to inspect, but what the numbers are doing is mostly opaque. Researchers can now trace small parts of a model's working, and what they find does not always match the model's own account of its reasoning.

So far, bigger has meant better. The computing power used to train the largest models has grown about five times a year since 2020. Whether that carries on is a real dispute, and much of what follows turns on it.

Part two

From talking to doing

A chatbot answers, and you decide what to do. An agent is the same kind of model given tools, such as a browser, a code editor or a company login, and told to get on with it.

It works in a loop: think, act, look at what happened, think again. It can go round that loop hundreds of times without a person checking.

The trouble with rewards

Stage three of training rewards success. But a computer can only reward what it can measure, and what you measure is never quite what you mean.

The classic case is a boat. In 2016 OpenAI trained a system to play a racing game called CoastRunners, rewarding it for points. It found a lagoon where three targets kept reappearing, and circled there for ever, crashing and catching fire, never finishing the race, and outscoring human players by about a fifth.

Part five

Why anyone thinks this ends badly

The case that AI could end humanity is not one claim. It is a chain of five, and each has to hold for the conclusion to follow.

This is the most useful thing to grasp about the whole row. The camps are not shouting at random. Each one attacks a different link. Press a link to see who disputes it.

So what are the odds?

People in this debate put numbers on their fear. The jargon is p(doom): the probability of catastrophe. Treat every figure below as a mood with a decimal point. Nobody has a method for working these out, the definitions of "doom" differ, and the people most willing to give a number are not a fair sample.

Part six

A field guide to the factions

The labels people throw at each other, such as doomer, booster, decel or grifter, tell you little. Two questions sort almost everyone.

How powerful will AI get, and how soon? And: how far should we hold it back? The map below plots the main voices on those two lines. Where we place people is our reading of what they have published. People move, and several would dispute their spot.

A short test

Spot the faction

Eight invented posts, written in the style of each camp. None is a real quote. See if you can tell who is talking. It is the skill this page exists to teach.

Part seven

The evidence, both ways

Most of what worries researchers comes from laboratory tests, many of them set up to tempt a model into bad behaviour. That is a fair way to find faults, and a poor guide to how often they happen in ordinary use. Until this summer, that was nearly all there was.

Part eight

What governments are doing

Take this with you

Five questions for the next row

There will be another incident, another letter and another pile-on. You will not have time to research each one. These five questions do most of the work.