A go-live for a new product is always a rewarding experience. Months of development and QA capitalize on the first customer adopting the product and its subscribers starting to use it. A “switch” in a load balancer -of some sort- is flipped and traffic starts flowing through your newly deployed solution. Engineers are happy that months of hard work is finally getting some real mileage. Sales is happy that revenue will be recognized soon. Professional Services are making travel reservations for the location of the next customer. And then …
And then all hell breaks loose. Stability issues that the QA process had missed are identified. Corner case scenarios that no-one could think off are run into and expose holes in the design or implementation. Customer issues are hard to reproduce in your own lab, so as to investigate and fix, requiring you to hack tests in a language you haven’t touched since ages ago. You pound at code with undocumented memory allocator settings. You find yourself reaching for the ancient scriptures, trying to troubleshoot and nail down an issue. You’re analyzing data from the now live systems in real-time, trying to gauge the magnitude of the problem, identify whether it’s a software or an integration issue. You find yourself working at post-midnight hours and still waking up at 06:00 am. You try to figure out workarounds till the hard at work developers manage to fix the underlying problems. You come up with aggressive acceptance test plans that optimally balance between the need for quality and quick turnaround. You operate under pressure, the customer requesting an immediate workaround in order not to rollback to the previously installed solution.
We have very good news. Both actions against the problem work as planned. This means that the highest priority issue of the customer is now solved.
You grab a cold beer. You can feel the lack of sleep creeping in your aging bones and mind, yet can’t really sleep. You remember a similar crisis a couple of years ago, with another product and another customer back at the time, you smile and think that I’m still not too old for this shit.
[Dedicated to Angie, Nikos, Petros and everyone else hard at work during the last 3 days; couldn’t have done it without you team]