On-Call Sucks

Being on-call can suck sometimes. Badly. How badly varies on a great number of factors hard and soft.

I just got off of a heavy on-call rotation. Nearly constant churn during hours of coverage. One bad part of that a lot of it was minor stuff. When big things go wrong, you feel like bringing you in was necessary. On small stuff, you feel increasingly frustrated while churning cycles on low-level crap. I did have one major failure...a SCSI interface board to an aging legacy storage array that housed a production cluster. Since we were fully redundant, and still had disk access for the cluster, it wasn't a major ordeal (other than starting the call at two in the morning). The worst thing about it was I was planning a HUGE storage-networking upgrade, involving the enterprise storage fabrics that was to start this week, and I kept getting dinged for small stuff.

All that aside, the on-call at my current job is infinitely better than other on-call duties I've had. For all the warnings I received from colleagues about how brutal this place could be, I've not seen it. In fact, it has been one of the most sane corporate jobs I've had since leaving employment at the University of Louisville. So I guess I should just be happy that the "heavy" on-call here is not soul-sucking, spirit-crushing, or asshole-filled...

So this morning was to be the beginning of my (our...it's not MINE after all) SAN upgrades. I'd spent weeks on the plans, and if we were given approval by the change-review board, I'd kick things off today and over the course of several days completely overhaul the storage network at the facility. We got approval.

Forty-five minutes ahead of when the vendor was the be onsite to do one portion of the work, I get a call from the engineer telling us there are numerous required upgrades to our SAN before we can install the two newest pieces of hardware. Nice. Known requisites, which the vendor never shared until the day of implementation. Argh. Not that I was mad at the engineer...he saved my ass by making me aware of a problem with our plans, and I could've royally screwed the storage for billions of dollars of hardware and transactions.

So the SAN upgrades go back to the drawing board, hopefully for implementation in early January. Blah. I feel stupid for not having grilled the engineer earlier about this, but technical review of the plans is supposed to happen before we pay for and receive anything from the vendor. Somehow that all got glossed over. Oh well. We'll get it installed, and in the meantime I can draft an even more detailed plan for it all, further reducing the risk of the work involved.

Such is the life of a geek.