Fault Injection
Fault Injection — Fault injection is the practice of deliberately breaking a mocked or real dependency—adding latency, 5xx errors, or dropped connections—to prove the system around it degrades gracefully instead of collapsing. It turns resilience from a hope into a testable property.
Definition
Fault injection is a testing technique where you intentionally introduce failures into a system to see whether it survives them. The idea sounds destructive, but it is the opposite of reckless: you choose the exact fault, the exact target, and the exact moment. What you are really testing is not the happy path—it is the recovery path.
In a developer-tools context, the most practical form is injecting faults into your mocks. When you mock a downstream API, you normally return a clean 200 with realistic data. Fault injection flips that: your mock now answers with a 3-second delay, a 503, or a connection that closes mid-stream. The component under test—your client, your SDK, your retry wrapper—is forced to do the thing it was built to do but rarely gets to practice: degrade gracefully.
The payoff is that you stop discovering resilience bugs in production. A missing timeout, a retry that double-charges, a circuit breaker that never opens—these are exactly the defects functional tests miss and exactly the defects fault injection surfaces in seconds.
Fault injection is closely related to chaos engineering, but they are not the same discipline. Chaos engineering usually runs broader, often in production-like or production environments, randomly terminating instances or injecting network partitions to challenge an entire system's assumptions. Fault injection is the narrower, more surgical building block—and when aimed at mocks, it is something you can run on every CI build without anyone's pager going off.
Why mocks are the cheapest place to start
Most teams already have a mock layer—WireMock, Mock Service Worker, a stubbed HTTP client, or a Toxiproxy between services in a test container. That layer is isolated, deterministic, and tear-down-able, which makes it the safest possible place to break things.
Unlike production chaos experiments, mock-based fault injection needs no blast-radius planning, no on-call approval, and no rollback script. You add a toxic to a proxy or flip a stub response, run the test, and delete it. The cost of entry is roughly one afternoon, not one quarter of platform work.
If you are new to resilience testing, start here. Prove that your timeout fires, your retry backs off, and your fallback renders—then graduate to platform-level chaos later if the system earns that investment.
Three faults you should inject first
Latency. Add 2–5 seconds of delay to a mock response. This exposes whether your client actually honors its configured timeout or silently hangs until the default 30-second socket timeout. The most common finding is a timeout that was never set.
Errors. Return a 500 or 503 for a percentage of requests. This validates retry logic—does it retry idempotently with a key, or does it risk a double write?—and whether your circuit breaker opens before the dependency fully melts down.
Connection resets. Close the TCP connection or kill the mock container mid-request. This is the harshest test: it simulates what happens when a dependency dies rather than politely declining. It catches missing dead-letter queues, lost transactions, and unhandled socket exceptions that no 5xx test will reveal.
What good fault injection proves
A fault-injection test is only useful if it asserts on behavior, not just on survival. After injecting the fault, check the things a real incident would care about: did the request time out within budget, did the retry use an idempotency key, did the user see a graceful 'processing' state instead of a hang, and did the transaction land exactly once?
That last point is the trap. A test that merely 'does not crash' can still lose or duplicate a payment. The discipline of fault injection is specifying the steady-state hypothesis—what should be true before, during, and after the fault—and failing the build when reality drifts from it.
Done consistently, this turns resilience from a hope written in a design doc into a property your pipeline verifies on every change.
Where it fits next to chaos engineering
Think of fault injection as the unit test of resilience and chaos engineering as the integration test of resilience. You do not pick one. You use mock-based fault injection in CI to catch the obvious defects early and cheaply, and you use chaos experiments to challenge the emergent behavior of the whole system under stress.
For a small team shipping a developer tool, the mock-first approach alone will catch the majority of painful outages. Reach for platform chaos tooling like Chaos Mesh or Gremlin only when your surface area and blast radius justify the operational overhead.