Because unavoidable accidents will still happen, and an autonomous car's behaviour in one is fixed by its programming — so the decision has to be taken before the situation arises, not in it.

The argument, set out by Alexander Hevelke and Julian Nida-Rümelin in their 2015 paper on offsetting human lives in unavoidable accidents, runs in three steps:

1. In ordinary driving, situations arise in which an accident involving personal injury has already become unavoidable — no available manoeuvre prevents someone getting hurt.
2. Fully autonomous cars, if they are ever licensed for public roads, will not be able to avoid such situations either. Better sensors and reflexes shrink the set; they do not empty it.
3. Therefore it must be decided in advance which maxims the car's behaviour should follow in them — because the car will do whatever it was built to do, and someone chose that.

The candidate solution the paper examines is a programming that, in an unavoidable accident, weighs potential victims against one another so as to minimise the number and severity of the expected injuries — the "injury-minimising" (verletzungsminimierend) approach.

The central themes that follow from it are the ones the rest of this topic works through: who is the decision-maker, the danger of instrumentalising the victims, the danger of systematically raising the risk for particular groups, the creation of perverse incentives, and questions of fairness.

Go deeper:

• TED-Ed: The ethical dilemma of self-driving cars (Patrick Lin) — four minutes that set up exactly this problem, including why the decision has to be made in advance.
• Ethics Commission: Automated and Connected Driving (Germany, 2017) — the twenty rules a national ethics commission actually wrote for this question (English PDF).
• Wikipedia: Self-driving car — the technology and its regulation, for the state of the world the argument assumes.

Responsibility for the impending accident, the number of people endangered, their age, and whether occupants get priority — and each one, once programmed, becomes a standing rule about whose life counts for how much.

* Each answer, once programmed, becomes a standing rule about whose life counts for how much *

Because vehicles must be programmed in advance for certain reaction patterns, society first has to work out which procedure would be the most justifiable. The open questions are:

• Responsibility for the impending accident — should someone who caused the danger bear more of it?
• The number of endangered people — do five count more than one?
• Their age — may a car prefer the young?
• Occupant priority — is it legitimate for a vehicle to give the safety of its own occupants precedence?

What makes the list contentious is not any single entry but what a decision on it is. A driver who swerves in a panic makes no judgement about relative worth; a rule that prefers the young is exactly such a judgement, applied to everyone, forever, by people who will never meet those affected. Discrimination that would be intolerable if announced as policy does not become acceptable because it is expressed in code.

There is a necessary corrective, though, which the paper insists on: the thing at stake — the loss of human life — is not peculiar to autonomous vehicles. Road traffic already kills. Increasingly automated vehicles merely force us to look seriously at that dark side of our traffic system, and to say out loud what we have so far left to chance.

Go deeper:

• MIT Moral Machine — judge the scenarios yourself, then see how your answers compare — the experiment that collected 40 million of them.
• Wikipedia: Moral Machine — what the study found, and the serious criticism that it is no basis for policy.