The engineers who design for failure first

A quiet shift inside backend engineering holds that production systems only earn their keep when every retry, duplicate and late arrival has already been accounted for in the design, not patched in later.

The 417 came back from a customer's procurement portal with the message "already used." The invoice had been delivered. Then it had been delivered again. Somewhere between an upstream publisher and a consumer that took longer to finish than its queue was willing to wait, two separate things had quietly gone wrong and produced the same symptom.

Apoorva Pindi, a Software Development Engineer at Amazon Web Services who works on the integration between cloud billing and enterprise procurement platforms, traced it to a pair of causes that only looked like one. The upstream system had published the same request twice inside roughly sixty milliseconds, and the messaging layer, seeing two different message IDs, waved both through.

Separately, the invoice processing step was taking about fifty seconds against a thirty-second visibility timeout, so the queue redelivered the message to a second worker while the first was still finishing. "Each component was doing the right thing in isolation," Pindi says. "Only together, they did something wrong."

That incident sits at the center of an argument she makes about how backend systems should be built in the first place. The conventional path, she says, is to design the ideal case and bolt on protection against failure once something breaks in production. The alternative, which she now applies by default, is to begin with the assumption that any message a consumer sees may arrive twice, arrive late, or arrive out of order, and to make that fact part of the design document rather than a later revision to it.

She wrote two designs for the invoice problem. One proposed a dedicated idempotency table in DynamoDB. The other moved the publisher and queue to a first-in-first-out configuration with deduplication built into the messaging layer itself. The team chose the second. There was no additional table to operate or reconcile against, rollback was straightforward, and the deduplication

identifier could include a second-level timestamp so that a genuine later retry would still get through while a true duplicate would be dropped. She also raised the visibility timeout to seventy-five seconds, enough margin that a slow run would no longer trigger a redelivery mid-processing.

Also Read: Chandigarh couple at centre of fresh US sanctions over helping Iran evade oil sanctions

The lesson she took, she says, was not about queues. It was about where failure handling belongs in the sequence of thought. "Before coding the happy path I think of failure cases such as retry, duplicate and timeout," Pindi says.

When she reviews code written by others on her team, that is the lens she uses. Engineers bring her their design documents before the wider review, and she reads for the paths that are not drawn. What happens on retry. What happens if a message appears more than once or in the wrong order. Whether a timeout on a downstream call is actually interrupting the work or merely postponing the acknowledgement. Her most common comment, she says, is a variation on the same question: the document assumes the message arrives once, so what happens if it does not.

She also asks writers to name the failure case they feel least confident about. By her account, that question tends to point directly at the gap worth closing.

The failure mode she sees junior engineers underweight most often is the duplicate request. The instinct is to treat an incoming call as a single, well-ordered event and process it accordingly. In systems where retries, redeliveries and timeouts are routine, that assumption breaks in ways that only surface under load. The remedies are not exotic: an idempotency key, a write that is safe to repeat, a timeout longer than the processing step it is meant to protect. What matters, she says, is that the questions are asked at design time rather than after a portal starts rejecting invoices.

Her definition of a well-designed system is unshowy. Retries are harmless. Failures are contained and visible. The next engineer can change something, or roll it back, without fear.

On the service she now owns end to end, the proportions have shifted. Document state versioning distinguishes stale, duplicate and fresh documents. Failed fanouts land in a retry queue backed by a dead-letter queue. On the operations emit path, a hardcoded hundred-millisecond timeout runs behind a fail-open pattern so that reporting cannot stall a customer's request. One detail she credits to a code reviewer: a coroutine-level timeout could not actually interrupt a blocking SDK call, so the timeout had to be pushed down to the client. The reviewer caught what the design had taken on faith.

What she wants to be, she says, is the engineer other people bring the hard reliability questions to. The route there runs through the paths that are not drawn on the diagram.