Pile of Dross

So I now have a Logitech Quickcam attached to my work PC for participating in videoconferences. And after some small adjustments (Hmm, having my colleagues’ mobile phone numbers, written on the whiteboard behind me, in shot probably isn’t a good idea) its now working.
I’m not sure this is a particularly good idea though - I’m (you may have noticed) not the most loquacious of folks and a lot of my emotional communication is non-verbal: facial expressions mainly. I’ve got the image capture window open while I’m reading e-mail and I really start scowling and staring wide-eyed when confronted by gross stupidity.

Also, I really need a haircut.

So after week 2, the best approach to achieving 150MB/s/day is to be on holiday for 4 days and not touch anything.

RAL T1 SC4 Status - GridPPwiki

So week 1 of SC4. Result: Hmm.

2 disk servers broken, expected rate not achieved, possible network bottleneck found. Pretty par for the course really. Its been a frustating week - not knowing whether our inability to achieve the rate we achieved in SC3 is due to some regression on our side or CERN’s (Probably both to some extent I think) but is the rate we’re achieving sufficently good that we should perseve with our current transfer method rather than switch to one which promises more effective use of resources but is less well tested and understood than the current one? The network issue was escpecially aggravating - there’s only one person in our team who really understands the network layout - its a closed book to the rest of us - especially when the 3 people who work in the office who’s whiteboard has the only diagram of the network, are out. And the disk servers? Well they’ve been standing idle for about month and then we started hitting them with instantaneous heavy transfer rates not really surprising they fell over.

Today was always going to be a complete bitch. The Tier 1 staff consists of 9 people - 1 manager, 2 experiment support people, 2 batch farm people, 2 disk server people and 2 Grid services people. Myself being one of the Grid people. This is an obviously gross simplification, but close enough. Today both batch farm people were out along with 1 disk server person and 1 grid person. We have a Monday morning review of last week, plans for this week meeting, and as we were waiting for the manager to turn up, one of the experiment people commented that if he didn’t turn up I’d be in charge. WTF!!!1! I’m pretty sure that both experiment support guys are senior in grade, even if I’ve been there longer than they have. However the manager did duly turn up so one small crisis was averted. You see this isn’t a good week for me to be in charge of anything, well really any week isn’t a good week, but especially this one. Because we’ve just started another Service Challenge, which basically involves the systems that I’m pretty much the sole admin of having data flung at them at high speeds continously. Which means that I’m pretty much living and dying with every bloody pixel on various RRD graphs. So my stress level is already quite high.

Still all seemed calm until about lunchtime when the remaining disk server person, became ill and went home. At which point I had a small breakdown at the realisation that I was the only sysadmin left in the team. Still things managed to continue without any major mishaps - until it became clear that the kit in the Service Challenge was achieving the rate we’d hope it would. One multinational video conference later and we had a plan, or at least something we had to try otherwise people would ask us why we hadn’t done it, important people - senior managment people. So somewhat exceeding my authority and risking incurring the wrath of our head batch farm person when he returns, I set 3 systems of the farm to drain - finish their currently running jobs and not start any more - and submitted a dns request get some new ip addresses setup, and also request X509 certificates for the systems too, all to implement this plan.

Okay, I can’t be bothered trawling my Inbox & Sent Folders to try and remember what I was up to last month - so no multiscreen acronym and abbreviation packed post to catch up. Breathe a sigh of relief or disappointment as you wish.