Archive for the ‘real work’ Category

Going live

March 18, 2010

A go-live for a new product is always a rewarding experience. Months of development and QA capitalize on the first customer adopting the product and its subscribers starting to use it. A “switch” in a load balancer -of some sort- is flipped and traffic starts flowing through your newly deployed solution. Engineers are happy that months of hard work is finally getting some real mileage. Sales is happy that revenue will be recognized soon. Professional Services are making travel reservations for the location of the next customer. And then …

And then all hell breaks loose. Stability issues that the QA process had missed are identified. Corner case scenarios that no-one could think off are run into and expose holes in the design or implementation. Customer issues are hard to reproduce in your own lab, so as to investigate and fix, requiring you to hack tests in a language you haven’t touched since ages ago. You pound at code with undocumented memory allocator settings. You find yourself reaching for the ancient scriptures, trying to troubleshoot and nail down an issue. You’re analyzing data from the now live systems in real-time, trying to gauge the magnitude of the problem, identify whether it’s a software or an integration issue. You find yourself working at post-midnight hours and still waking up at 06:00 am. You try to figure out workarounds till the hard at work developers manage to fix the underlying problems. You come up with aggressive acceptance test plans that optimally balance between the need for quality and quick turnaround. You operate under pressure, the customer requesting an immediate workaround in order not to rollback to the previously installed solution.

We have very good news. Both actions against the problem work as planned. This means that the highest priority issue of the customer is now solved.

You grab a cold beer. You can feel the lack of sleep creeping in your aging bones and mind, yet can’t really sleep. You remember a similar crisis a couple of years ago, with another product and another customer back at the time, you smile and think that I’m still not too old for this shit.

[Dedicated to Angie, Nikos, Petros and everyone else hard at work during the last 3 days; couldn’t have done it without you team]

Advertisements

Pushing for better

January 10, 2009

There comes a time when one has to stop being sentimental and start treating his day job as just this: his day job. It may be quite hard to do, especially when you’ve grown that much in your job but when circumstances change, people should change as well.

It’s now been a little more than a 3.5 year ride for me. The first 2.5 it was a very hard but at the same time a very enjoyable ride. Not that there weren’t bad or difficult moments but the nice ones were a lot more. And I really grew during that time. But the last year was a series of constant disappointments. From a career advancement perspective there was only a single really good moment, but this soon got clouded by two other bad ones, that made the original moment feel not really worth it (let alone other moments before it). And while this year I learned the hard way that things don’t really work out the way you like them and that you should be patient in such events and wait for the wheel to spin a little, there is only so much one can take.

Yesterday I decided to move the pieces. There is no point in continuing to perform a function that doesn’t fulfill you, so I’ve decided to make sure that changes. In addition, TINSTAFL so there is no point in feeling like you offer others a free lunch (or one at a significant discount) either. It remains to be seen how it will turn out …

The bigger picture

January 8, 2009

Perforce is cool. Really cool. Sure it may be centralized, require a network connection all the time and have a strange commit flow (yes p4 open, I am looking at you). However the GUI client is fantastic, the CLI tool excellent (with well thought-out command names), p4 help almost always saves the day, the documentation is great, the technotes can be a life savior. And to add to that they offer a proxy. A perforce proxy is no substitute for a distributed SCM (since in turn it must be always connected to the main server), does not automatically mean that user experience is identical as if the main repository was on the local LAN but is a welcome add-on.

Lately people noticed our perforce proxy being rather slow. Admittedly it’s an old machine, with a measly (for today’s standards) 512MB RAM. So what was the verdict? Let’s buy a new one! YEAH! And let’s make sure that it can expand to 32GB, the entry level model that grows to “just” 8GB are probably not enough.

I am frequently impressed by how often people end-up getting confined in their micro-world and fail to get the big picture. Good product QA engineers, who could be great if they would get a better understanding of the underlying O/S. Same for development engineers. Customer support engineers who have an excellent grasp of the customer and provide prompt responses for documented issues but with poor troubleshooting skills, resorting to product development engineers for the slightest of problems (this is not an excuse for poor documentation btw). And the list goes on.

For what is worth, per the Perforce technical notes 8GB would probably be a little bit too much for the new server. 512MB would be more than enough. With a careful setup (disabling unnecessary daemons of the underlying O/S) even 256MB could suffice (and yes this means that we may not need to buy a new server)

On usability

January 8, 2009

You can tell that the usability of a system has a serious problem when a very simple task takes at the very list thirteen arcane commands in a Unix prompt. You can also tell that you need a usability expert/committee of some sort when this problem is there for many years (if not more) and noone has done something about it.

P.S. Just because there isn’t a huge market around them (like web sites) doesn’t mean that command line tools cannot be (un-)usable.