There is a truism that I face in my day job, it is not unique to my day job.
By the time I am assigned a job, it has passed through perhaps 20 individuals, all of whom work for 3 or 4 or 5 resellers of resellers and channel partners and all the other modern business lingo for everything with is not 100% “in-house”… which is 100% of everything nowadays.
In this passing through process, human nature being what it is, people only see, and read, and focus on, and then repeat and pass on, that which makes sense to them, that which they can understand, that which they grep.
All else is discarded.
“All else” being another way of describing “100% of what I, or any other engineer or technician involved in the job, actually fucking needs to know.”
So a fast food / drive in joint that can only do cash sales and cannot do proper stock ordering or staff payroll or anything else, because the internet connection has been down, for a week now and counting.
- I’ve been on-site three times, nothing that I can fix or touch, eg the router itself, has a problem.
- BT infrastructure guys have been on site twice, nothing that they can touch or fix, eg the cable from the street box to the master socket on the wall inside the premises, has a problem.
- The problem exists in the structured cabling between the router in the wall cabinet on one side of the office, and the BT master socket on the other side of the office.
- The company / site itself, well they have different organisational departments, for example an IT department, a site structured cabling and infrastructure sub-contractor, a site IT hardware sub-contractor, this prevents the site from having to employ and then send their own staff to site to fix anything.
- The site itself employs a dozen staff and it is a busy / popular fast food joint that turns over thousands a day, or used to, when they were able to process plastic transactions, when their internet worked, last week
- The site itself realises, sic, how important IT is to trading, this is why in all these outsourced and sub-contracted services they have SLA’s that guarantee things like 4 hour response time and so on, there is a whole paper chase of this shit extending through several layers of interconnected resellers and channel partners.
- The site has still been down a week, and no actual penalties have been triggered, because everyone has done their job, on paper, “best efforts” and all that, we are all only responsible for the small slices of pie our contracts cover, individually.
- When it works, the whole site hangs on one now obsolete model of cisco xDSL router that you can buy used on ebay for ten bucks, and one bog standard xDSL line that is technically the same as the bottom of the range cheapo entry level domestic xDSL connection, but at business prices, which is why they have managed to get a BT engineer to physically attend site twice, promptly.
A second backup xDSL or other line/connection isn’t on the cards, in this nickel and diming world, such expenditure is a “waste”. Who needs it, we have all these SLA’s for the single xDSL line anyway…
So, cisco console commands…
XTU-R (DS) XTU-C (US) Chip Vendor ID: 'BDCM' 'GSPN' Chip Vendor Specific: 0x0000 0x0010 Chip Vendor Country: 0xB500 0xFFB5 Modem Vendor ID: 'CSCO' ' ' Modem Vendor Specific: 0x4602 0x0000 Modem Vendor Country: 0xB500 0x0000 Serial Number Near: FCZxxxxxx 887VA-M- 15.1(4)M Serial Number Far: Modem Version Near: 15.1(4)M Modem Version Far: 0x0010 Modem Status: TC Sync (Showtime!) DSL Config Mode: AUTO Trained Mode: G.992.5 (ADSL2+) Annex A TC Mode: ATM Selftest Result: 0x00 DELT configuration: disabled DELT state: not running Trellis: ON ON Line Attenuation: 36.0 dB 18.5 dB Signal Attenuation: 36.0 dB 11.0 dB Noise Margin: 19.8 dB 11.0 dB Attainable Rate: 11816 kbits/s 984 kbits/s Actual Power: 22.2 dBm 10.4 dBm Total FECS: 0 0 Total ES: 0 0 Total SES: 0 0 Total LOSS: 0 0 Total UAS: 0 0 Total LPRS: 0 0 Total LOFS: 0 0 Total LOLS: 0 0 Bit swap: 0 5
This is fairly routine stuff, and it’s the bits in purple you always look at, because attenuation is a measure of line quality, anything over 45 and you start to get issues, ideally it should be under 20, but it never is… every single joint in the copper adds to the attenuation, and so attenuation should be as low as possible, and noise margin should be as high as possible. This isn’t from the actual site in question BTW, these numbers are poor, but it all works fine anyway.
So I can tell, when I go on-site, the problem isn’t the cisco router, obsolete and old as it is, but what the fuck, you guys paid for it, and it proves a point, so I’ll swap it out anyway, no change, exact same problem still exists, the line flaps and drops more than it stays up.
It’s what we will call a “copper fault”
I’ll note all this on the paperwork, and forward it up the channels.
The BT guy turn up, and does his job, notes everything on the paperwork, and forwards it up the channels.
Between us, the I’ve just done the router, and found no issues, so suggest a copper fault, the BT guy turns up, tests the copper between the street box and master socket, finds no issues, notes it must be a structured cabling issue between the master socket and router, I’m sent back to site a second and third time, because apparently BT have tested the line and there are no issues.
You see, that very first bold underlined red text bit up top, in this passing through process, that’s where the issues lie, I can do my job, the BT guy can do his job, but all the pen pushers in between only forward the bits they can see, the bits they can understand, and structured cabling is just two words to them, as indeed are line attenuation, noise margin, carrier detect, dsl showtime, attainable rate, eg the entire fucking lexicon…. moreover they are two words that sound sort of techy-ish but not very important.
What’s really important is the service level agreements and contractual obligations and customer downtimes and keeping your line manager happy happy happy, by dealing with the job on your desk so that it is no longer on your desk, it’s closed and moved on to someone else’s desk or department.
See the bold green text above…. oh yes…
And if you think this is in any way a unique or unusual or unrepresentative job, think again.
And if you think this is just because of the level I work at, think again, less than 24 hours before this I was working on some very high end, dare not name either the kit or the location, stuff, we are talking backbone stuff, 250,000 fibre customers going into one switching card, and I’m the one swapping the main card while we flip to the backup one for an hour and then flip back and test, zero service interruption… and this was just one card in one rack, together with the adjacent racks you’re looking at 20-30% of the entire UK fibre network and business and residential internet / television / phone and other data.
At *that* level, there is nobody in the system who does not speak and understand technical / engineering language, and we all double check each other, especially if talking on the phone, yes mate, I can see the link lights have gone out, can you conform your end, rack 3 core 3 fibre card 2, confirm? I confirm, you are good to go and pull rack 3 core 3 fibre card 2. Thank you, I will call you back in ten when we have done the swap. etc
*I* still get paid the same (shit..lol) rate per hour, no matter the job, whether I am in the UK backbone or skidding around on the greasy floors of a local fast food joint, it’s all the fucking same to me.
The above “passing through process” is a cautionary tale for anyone even remotely concerned about the state of western civilisation, google the Ebola clipboard guy if you like, it’s symptomatic of exactly what I am talking about here, and it is ubiquitous and universal.
And these are the “somebody” you’ll all be relying on, as in “somebody” will step in to prevent this disaster scenario from happening, when the reality is they are the water at the base of the glacier, lubricating everything, ensuring that the disaster scenario cannot be avoided.
