What shipping something broken taught me

A few months into HostLink I pushed a deploy on a Friday afternoon that changed how booking confirmations got sent, and I only tested the happy path. It worked fine for about four hours, then a third-party email provider had a brief outage, and instead of retrying or queuing, the new code silently dropped the confirmation and marked the booking as sent. Nobody got an error. Nobody got an email. I found out Monday morning from a customer who’d shown up to a booking the other side didn’t know existed.

The bug itself wasn’t the interesting part, that’s a normal failure mode for anyone shipping features fast. What I actually learned was about my own reaction to it, and about what my process was missing.

My first instinct was to explain it away internally — the third-party outage was outside my control, it was a rare timing coincidence, this wouldn’t normally happen. All true, and none of it mattered, because the actual failure was that I’d shipped code with a silent failure path at all. An outage happening was inevitable eventually; the code having no way to notice and surface that outage was the actual bug, and it was entirely mine.

The second thing was about deploy timing, which I’d never taken seriously before. Shipping on a Friday afternoon meant the failure window landed on a weekend, when I wasn’t actively watching anything and the customer had no fast way to reach me. I don’t think Friday deploys are inherently wrong, but I do think shipping anything that touches a critical path — payments, confirmations, anything where silence looks the same as success — right before a period where you won’t be watching it is a bad trade, and I was making that trade out of habit rather than a deliberate choice.

What changed afterward: any code path that can fail now has to fail loudly, because “unlikely” and “silent” together is how a rare failure turns into a bad Monday. I also added a basic monitoring check for this shape of problem, a confirmation marked sent with no corresponding delivery event, which would have caught it within the hour instead of over a weekend. And I stopped treating deploy timing as a non-decision. It’s a real tradeoff now, not autopilot at 4pm on a Friday because that’s when the feature happened to be done.