What is Chaos Monkey, and why would anyone deliberately break their own production systems?

A Netflix tool that randomly shuts down virtual machines or containers during business hours. Failures then happen constantly, so developers are forced to build in redundancy and automatic recovery.

The reasoning goes back to the principle that "failure is normal". In a large cloud, instances will die. The question is only whether that happens at 3 a.m. during a real incident, when the team discovers the service can't cope, or at 2 p.m. in a controlled way, while engineers are at their desks and can fix the weakness.

Chaos Monkey chooses the second. Because any instance can disappear at any time:

• no service can rely on one particular machine staying up,
• every service needs redundancy and automatic recovery,
• weaknesses show up early and in daylight.

It runs during business hours on purpose, so that people are around to respond. This practice of deliberately injecting failures to prove resilience is called chaos engineering.

Go deeper:

• Chaos Monkey (Netflix) — the tool's own docs: random instance termination in production.
• Principles of Chaos Engineering — the method behind it, written by its practitioners.
• Chaos engineering — Wikipedia — history, the Simian Army and later tools.