Pile of Dross

Today was always going to be a complete bitch. The Tier 1 staff consists of 9 people - 1 manager, 2 experiment support people, 2 batch farm people, 2 disk server people and 2 Grid services people. Myself being one of the Grid people. This is an obviously gross simplification, but close enough. Today both batch farm people were out along with 1 disk server person and 1 grid person. We have a Monday morning review of last week, plans for this week meeting, and as we were waiting for the manager to turn up, one of the experiment people commented that if he didn’t turn up I’d be in charge. WTF!!!1! I’m pretty sure that both experiment support guys are senior in grade, even if I’ve been there longer than they have. However the manager did duly turn up so one small crisis was averted. You see this isn’t a good week for me to be in charge of anything, well really any week isn’t a good week, but especially this one. Because we’ve just started another Service Challenge, which basically involves the systems that I’m pretty much the sole admin of having data flung at them at high speeds continously. Which means that I’m pretty much living and dying with every bloody pixel on various RRD graphs. So my stress level is already quite high.

Still all seemed calm until about lunchtime when the remaining disk server person, became ill and went home. At which point I had a small breakdown at the realisation that I was the only sysadmin left in the team. Still things managed to continue without any major mishaps - until it became clear that the kit in the Service Challenge was achieving the rate we’d hope it would. One multinational video conference later and we had a plan, or at least something we had to try otherwise people would ask us why we hadn’t done it, important people - senior managment people. So somewhat exceeding my authority and risking incurring the wrath of our head batch farm person when he returns, I set 3 systems of the farm to drain - finish their currently running jobs and not start any more - and submitted a dns request get some new ip addresses setup, and also request X509 certificates for the systems too, all to implement this plan.

Trackback This Post | Subscribe to the comments through RSS Feed