Pile of Dross

Today was always going to be a complete bitch. The Tier 1 staff consists of 9 people - 1 manager, 2 experiment support people, 2 batch farm people, 2 disk server people and 2 Grid services people. Myself being one of the Grid people. This is an obviously gross simplification, but close enough. Today both batch farm people were out along with 1 disk server person and 1 grid person. We have a Monday morning review of last week, plans for this week meeting, and as we were waiting for the manager to turn up, one of the experiment people commented that if he didn’t turn up I’d be in charge. WTF!!!1! I’m pretty sure that both experiment support guys are senior in grade, even if I’ve been there longer than they have. However the manager did duly turn up so one small crisis was averted. You see this isn’t a good week for me to be in charge of anything, well really any week isn’t a good week, but especially this one. Because we’ve just started another Service Challenge, which basically involves the systems that I’m pretty much the sole admin of having data flung at them at high speeds continously. Which means that I’m pretty much living and dying with every bloody pixel on various RRD graphs. So my stress level is already quite high.

Still all seemed calm until about lunchtime when the remaining disk server person, became ill and went home. At which point I had a small breakdown at the realisation that I was the only sysadmin left in the team. Still things managed to continue without any major mishaps - until it became clear that the kit in the Service Challenge was achieving the rate we’d hope it would. One multinational video conference later and we had a plan, or at least something we had to try otherwise people would ask us why we hadn’t done it, important people - senior managment people. So somewhat exceeding my authority and risking incurring the wrath of our head batch farm person when he returns, I set 3 systems of the farm to drain - finish their currently running jobs and not start any more - and submitted a dns request get some new ip addresses setup, and also request X509 certificates for the systems too, all to implement this plan.

Okay, I can’t be bothered trawling my Inbox & Sent Folders to try and remember what I was up to last month - so no multiscreen acronym and abbreviation packed post to catch up. Breathe a sigh of relief or disappointment as you wish.

For somewhat dumb reasons (not mine - I’m the smug smartarse who pointed out failing of logic, but anyway) I now have a Gizmo account - apparently our site networking folk prefer(allow) it to Skype. And it integrates with Adium, my current IM client, as a bonus.

Chuck me an e-mail if you want contact details.

Look like the Register got their copy of one the Sc3 press releases. Interestingly none of them mention that these latest results are from a rerun that had to be done because the first set was so bloody awful :-), and that we should have been at this stage about 6 months ago. Note also the blatant attempts in the quotes to appear relevant to the big shindig they’re all at in India at the moment.

I’m not going to say anything more about it - just point out the RAL Log for SC3 and its change history

Yay, success. Overnight we were accepting data from CERN at a rate of 150MB/s. This is an important number because its the nominal rate we’ll have to accept data at when the LHC switches on, though spot data rates my be higher - up to 600MB/s has been estimated. Also, this is really the first time we’ve seen sustained rates of over 1Gb/s on the optical private network we have to CERN, despite it being provisioned at 2Gb/s.

Another impressive feat is that last week we had 12 sites accepting data at 800-1000MB/s cumulatively, yesterday and today with the number of sites trimmed down to 6, those sites (us included), were getting close to accepting data at 800MB/s. A far cry from the summer when with 12 sites it was hard to sustain 600MB/s.

The atmosphere’s different here too, in the summer it was desperate, we were swamped with software problems and hardware glitches, this time its subtly different - we’re still seeing some hardware glitches and some software behaviour we don’t understand, but they’re puzzles to ponder, not horrendous obstacles to overcome, that doesn’t mean I haven’t spent times trying to raise network rates by thought alone, by peering intently at the network graphs - just that it seemed to work this time :-D.

Ha, last night I was moaning to myself about a file downloading at 2.3kb/s being too slow, today I’m moaning to myself about our network traffic only being 800Mb/s.

World’s biggest grid seeks secrets of the universe

Silicon has a go at explaining what I do. Though they do seem to have got confused between Tier 2 sites in general and some Tier 2 sites participating in a particular aspect of the whole shebang.