Break it on purpose, before it breaks on you
Chaos engineering is the discipline of experimenting on a system to build confidence that it can withstand turbulent conditions in production. This page covers the principles, the experiment loop, common faults to inject, and where to go next.
Start at the centre. Widen the blast radius only as confidence grows.
The experiment loop
Every chaos experiment follows the same five steps.
- 1
Define steady state
Pick measurable signals of normal behaviour, such as latency, error rate or throughput.
- 2
Form a hypothesis
State what you expect to stay true when something fails.
- 3
Inject a fault
Introduce a real-world failure on a small, controlled scope.
- 4
Observe
Compare the system against its steady state using your telemetry.
- 5
Learn & fix
Close the gap, then widen the scope or run it again automatically.
Five principles
From the Principles of Chaos Engineering.
Build a hypothesis around steady state
Focus on measurable output, not internal details.
Vary real-world events
Crashes, slow networks, bad deploys and dependency outages.
Run experiments in production
Where real traffic and real configuration live, once you are ready.
Automate to run continuously
Manual experiments do not keep up with change.
Minimise the blast radius
Limit the impact so a failed experiment does not become an outage.
Faults worth injecting
Start with the failures your system is most likely to meet.
- Pod and instance killsDoes the service recover when a replica disappears?
- Latency and packet lossDo timeouts and retries behave under a slow network?
- Network partitionsHow does the system split, and how does it heal?
- CPU and memory stressWhat happens at resource limits?
- Dependency outagesDo fallbacks work when a downstream service is gone?
- Bad configuration or deploysCan you detect and roll back quickly?
Related on this site
Tools, reliability practice, testing and platform pages.
Chaos engineering tools
Chaos Engineering ToolsA directory of chaos tools for Kubernetes, containers, cloud and distributed systems.Chaos Engineering in KubernetesChaos tools built for Kubernetes: pod termination, scheduled chaos and more.Reliability & operations
Site Reliability EngineeringThe SRE hub: reliability engineering, tools and service meshes.Reliability EngineeringPrinciples and practice for highly available, fault-tolerant systems.SLO & Error Budgeting ToolsSet the targets that chaos experiments then test.Testing
Platform & runtime
Tools & references
Official sites for the projects and services mentioned above. Opens in a new tab.
Questions
Short answers.
What is chaos engineering?
Chaos engineering is the discipline of experimenting on a system to build confidence that it can withstand turbulent conditions in production. Teams inject controlled failures and watch whether the system keeps meeting its goals.
Is chaos engineering the same as testing?
Not quite. Conventional tests check known conditions with expected answers. Chaos experiments explore how a running system behaves under unexpected failure, including interactions between services that tests rarely cover.
Do I have to run it in production?
The principles encourage production because that is where real traffic and configuration are. Most teams start in staging, keep the blast radius small, and move to production as confidence and observability grow.
How do SLOs relate to chaos experiments?
SLOs define what normal looks like. A chaos experiment checks whether the system stays within its SLOs while a fault is injected, which makes the result easy to judge.
What do I need before I start?
Good observability, a clear steady-state definition, an easy way to stop the experiment, and agreement from the teams affected. Start small, such as killing one replica of a non-critical service.