I’ve been busy of late, hence the dearth of new posts..
My current gig is basically summed up thus, world + dog are chasing economies wherever they can find them (a good example is regional offices that years ago would have been on leased lines now being connected by xDSL) and so ACME corp’s 447 regional offices get new Cisco 887 adsl routers and all that, and the IT management can then be outsourced and offshored…. 447 expensive leased lines dropped, the in house 500 strong IT department sacked en masse, loadsa money saved, trebles all around at the bean-counters offices.
But some cunt has to turn up with the box and physically plug in the patch cables and so on, and when, not if, when that shit breaks, some cunt has to turn up and physically reset or repair the thing that cannot be fixed remotely…. even if that someone is just a remote pair of hands for a resellers resellers resellers reseller….
Don’t get me wrong, this shit is slick, but it is a basic engineering principle that the more layers of complexity you build up, the more there is to go wrong… which is why twice in the last week alone the NatWest Bank customers have seen all the ATM‘s simply stop working and no on-line banking either, and this is being repeated across the nation in all things IT.
Like the song says, Do your fucking job till the end
Till your job ends that is…. meanwhile back at the gig the crowd I work for are all gung ho, gangbusters and corporate image, which is fucking great while it lasts, which is by definition going to be a finite amount of time, we are hyenas feeding on corpses, for the moment it is a banquet…
I smile sweetly at them all, and friday rolls around and I think to myself that is another week’s money grabbed, wonder what next week will bring, because you see I am old enough and cynical enough to know that in this solyent green world, the crowd I work for can disappear with as little warning as the jobs of those we are replacing with little Cisco boxes (themselves now made in the Czech republic, oh the irony) went down the swannee…
They tell me about all the valuable skills and qualifications I can earn while working for them, and there is an element of truth in that, but I had valuable skills and qualifications in my previous trade of marine engineering, and they don’t put food on the table today, but my survivalist attitudes to life do, so what is more useful to me?
Never take a job you aren’t prepared to walk away from on a moment’s notice is a good motto, because already in this young and dynamic company I can see signs of the rot setting in, and the infection is spreading a lot faster than it did 20 years ago… I see this all over now, they will give some guy a £25k car to turn up at the customers premises in, because it looks good, but no 5 dollar uniform sweatshirt, just wear something of your own, and if you are given any tools they came from walmart, it is utter fucking madness…. exactly the same brand of utter fucking madness that created the jobs in do in the first place… by sacking all the IT staff and sticking in a remotely managed router and some switches.
Of course VOIP is all the rage, so when the cute little cabinet goes down the ACME corp regional office does not just disappear from the HQ WAN, all the fucking phone lines go down too… how many of these sites have all this shit running on a UPS, even a cheap and nasty will only keep it running for ten minutes SOHO job from APC or similar?
You got it, haven’t seen a single fucking one yet….
It is fucking dreadfully incompetent and amateurish, I don’t give a flying fuck how swish and fancy and cute all this remotely managed Cisco kit is, WHEN IT IS WORKING, I don’t care how impressive the tricks are that you can do, WHEN IT IS WORKING, I don’t care with what ease you can do quite complex tasks, WHEN IT IS WORKING, all I see is a system that studiously ignores the 9,000lb gorilla in the room, what the fuck do you do when it stops working, and their answer to that is to point at dudes like me…. whoooosh…
So anyway I’m chilling after a job yesterday with another of the field engineers, who is of a similar age to me, and we are discussing this, and the one thing my extensive experience has taught me…. and this is from the year dot of web servers on…
- The least likely person to bork such a box is the field engineer sat there physically in front of it, in true CYA mode he covers his ass at every step, when I am asked to type in console commands for a box that has lost connectivity to a remote IT management centre I read and spell everything back phoenetically, and then ask them, do you want me to press return now? No matter how simple the command.
- The MOST likely person to bork such a box is the remotely connected tech telnetting in or whatever, they don’t give a fuck, and this is before they get confused between the three other field techs they are talking to simultaneously to me.
- The MORE of a wizard the remote tech is, the WORSE they will bork the box…
All of which means that instead of us field guys being remote waldoes for the megamind remote admin guys, which is how all this shit is marketed by the bean-counters, we are just another point of failure, for exactly the same reasons that someone playing Call of Duty will have a different approach to a crunchie on the ground in Afdiggastan with actual bullets flying around…
Now these people, if you push them, will admit that there are things like the graphic above, a “cascade failure”, but these same cunts have never had to RECOVER from one, because the fact is they have never been in one, of if they have, they were but one node…
I can distinctly remember being in a large hydroelectric turbine hall when a (local) cascade failure hit, because one of the turbines was tripped out by a vibration sensor, which they think was caused by a log getting down into the vent, so one goes down, and it takes aaaaaages to spin down, but the SOUND is indescribably different when it is not under load, and then the next one went because it was overloaded thanks to the first one going down, and then then remaining three went almost together…. and everyone is stood there looking at each other and the hall lighting goes out, and emergency DC lighting flicks on and the turbines continue to spool down… it is the most eerie motherfucking experience… and it took on the onsite diesel gen set and four hours or work before they could start spooling up again, another two hours to get the first two turbines synced to the grid, and another four hours for the remaining three.
But they had ENGINEERS on site, not fucking remote wizards and the only thing on-site some field techs told on a phone press this button now, now press this one, now type this in, now move that cable from here to there, OK I’m in, you can go to the next job ta….
SLA’s, well SLA’s are fulfilled if the resellers reseller can get a warm body on site within 4 hours, that warm body doesn’t have to actually DO anything, or FIX anything, he is just there so the SLA penalties can’t be invoked.
What the people I am currently working for do not know, that I do, is this.
THE DIFFERENCE BETWEEN THEORY AND PRACTICE, IS GREATER IN PRACTICE THAN IN THEORY.
So what happens in extreme cases, well someone ships a new box down, and it gets swapped out and we see if that fixes the problem, the only thing rarer than a UPS is the proverbial “smoking gun” when responding to an error call, nobody know what went wrong or what the causes were, and nobody gives a fuck, this job has had a 2 hour slot allocated to it, and that’s all there is.
Various three letter government agencies are waffling on about the threat of cyber terrorism, and hackers are getting sent to gitmo for 999 years of waterboarding pre trial, but the fact is that the real terrorists are all the fucking beancounters putting these bastard systems in place in the first fucking place, it isn’t IF it falls over, it is WHEN it falls over.
Currently these failures ain’t that bad, wossname bank goes down for 6 hours, wossname ISP goes down for 8 hours, wossname supermarket goes down for 4 hours, but no measures are being put in place to improve on this, on the contrary…. the opposite is what is happening.
Currently, cascade failures in IT have been confined to so called fucked up countries where fucked up stuff like the so called arab spring uprisings were going on, and again shit was blamed of guvvmint shutting shit down, hasn’t YET happened to a western country on the scale of the seventies east coast USA power grid cascade failure, which was ultimately caused by ONE part dying… hasn’t happened YET.
But it’s gonna, why else is everyone getting the pre – emptive bullshit excuses in place about digital pearl harbours.
And it is not just ACME corp and your local supermarket and your local mobile phone shop doing this shit, it is also your local Court of law, your local Police station, your local lawyers, your local bank, your local hospital, and the technology is spreading in all these places.
Sure, they may well have a diesel genny out back that can be fired up to keep the lights on, but what fucking use is that when packets carrying everything from data to voice suddenly find no routes outside the LAN?
Which reminds me, next week’s money I need to buy myself a new NAS box and a couple of WD Red 3tb disks… lol
