How do RTOSes prevent priority inversion, and what happened on the Mars Pathfinder mission?
With priority inheritance: while a low-priority task holds a lock that a high-priority task is waiting for, it temporarily runs at the high priority, so medium tasks cannot preempt it. Pathfinder kept resetting on Mars in 1997 until this was enabled on the mutex involved.
Priority inheritance fixes step 4 of the inversion. The moment Task 1 blocks on the lock held by Task 4, Task 4 inherits Task 1's priority. Task 2 can no longer preempt it, so Task 4 finishes its critical section quickly, releases the lock, drops back to its own priority, and Task 1 runs. Task 1's delay is now bounded by the length of Task 4's critical section only.
Alternatives:
- Priority ceiling: a lock carries the priority of the highest task that ever uses it, and whoever takes it runs at that priority.
- Design: keep critical sections short, and avoid sharing resources across very different priorities.
In FreeRTOS this is the practical difference between a mutex (which implements priority inheritance) and a binary semaphore (which does not). Use a mutex to protect a shared resource.
Mars Pathfinder, 1997. Days after landing, the lander's computer began resetting itself. A low-priority meteorological task held a mutex on the shared information bus, a high-priority bus-management task waited for it, and medium-priority communication tasks kept preempting the low one. A watchdog saw the high-priority task miss its deadline and reset the system. The VxWorks mutex had a flag for priority inheritance that was switched off; engineers reproduced the bug on a replica on Earth and turned it on by remote patch.
Go deeper:
Glenn Reeves — What really happened on Mars? — the Pathfinder software team lead's own account of the diagnosis and fix.- Wikipedia — Priority inheritance — the protocol and its limits compared with the priority ceiling protocol.