“I also want to express my appreciation at being able to share my telephony passion with someone who actually understands what I’m talking about. I hear all the time, people want to be able to walk up to a telephone, pick it up, dial and it just works. They don’t care about the why or how it works. They don’t wax eloquent about the beauty of a route pattern, or think long and hard about survivability, or stop to ponder why things do what they do, and what historical precedent may be contributing to the current behavior.” Recent private email with Jason
Today’s rants is all about the “consumer”, those pesky “end users” and how those same “consumers” fail to understand IT, IS or telephony and blame the boogy man of cable companies for not responding to calls on a very unusual series of events.
If you were one of the lucky victims of loosing Comcast Cable TV today (markets impacted were Sacramento, San Francisco, Houston, Denver, Detroit, Boston/Manchester, and other markets), you’re not alone. Supposedly Comcast had a breakdown of transmission of several cable networks today. It was resolved within a couple of hours on my Comcast (formerly Adelphia) headends by noon. Unconfirmed reports claim Internet services were down in some regions. I didn’t notice today, I had my set off, until my grandmother was telling me she couldn’t get the Fox News Channel, and assumed maybe was an outage on the Newscorp stations. I then did a search to find out, oh yeah there was an outage. A big one.
Regardless my mother had tried to call Comcast and we got “All Circuits are busy” announcement followed by a PSTN-equivalent reorder tone. My mother never heard such thing in years, probably in the days when we still had SXS switching in the 603 area code. (ESS wasn’t fully available till 20 years ago!) I called my cell to see if the modem from 2010 was acting up again. I heard my iPhone go off.
The problem is people are moaning and b-tching about “I lost my cable and my call gets dropped” blah, blah blah.
The problem is that Comcast gets a load of calls to begin with. I bet nearly 70% of the Internet related issues is because the Internet is just a complicated protocol and because the router lives inside the home, per to Part 68 like rules, the ISP really doesn’t want to deal with keeping the modem/router/gateway reliable – it’s the customer’s responsibility.
Today was a problem in itself. The Cable TV services that typically doesn’t fail. For me, I was able to get my local stations plus the regional sports networks and ESPN, etc. (Most of my local stations broadcast through fiber optic lines to the cable providers. Believe it or not, the telecommunications standards of using fiber over dishes is becoming a standard fare.)
Because of this very unusual abnormality, the lines were loaded. I know we have a major call center out of Manchester for north of Boston customers, and I believe there are other regional call centers that answer calls to the 1-800-COMCAST. As I previously describe in the definition of the PBX, the forumla with ACD PBX systems is lets say there is 100 person call center, and you add 100 lines, then have about 50 to 70 more to have overflow or to “queue” calls, and enough lines to escalate to Tier 1 to 4 support centers. No call center in the world is designed to take every call in the world at the same time, because you don’t have enough people to answer and/or you would have to many people in the jail cell waiting to be answered.
Also noticing the NBC Universal side of the business going to Cisco (after being a loyal Avaya Red customer since the time of Divestiture), I believe Comcast/Xfinity have used Cisco even before they bought NBCU. Imagine the call center using Cisco Call Manager (the PBX/server version) and there is a bunch of overloaded calls coming into the CallManager. The system more than likely will crash depending on how much load the server can take at a time (roughly a CallManager can take about 4000 users among a series of servers.) If there was so many outages on the Cable TV side that flooded their call centers, I’d suspect the callmangers started to fire up and when these things crash and burn, it’s not a Definity where it can boot up within 5 to 10 minutes at a maximum. This could take up to 30 minutes plus the time to reboot the hardware (phones and gateways) and rebooting some Cisco routers or gateways takes a real long time. But again how often would this happen?
This is one of the deadly sins of Voice over IP. It is designed to be a fancy application running on a PC. PCs are a godsend if it works properly. PCs weren’t designed to handle telephony by design. Over the last decade it’s gratefully improved, but it is what it is. And depending on the hardware setup (SSDs vs magnetic hard drives, if RAID is used or if it’s a virtual machine is there enough resources) can be a factor.
Another thing to add is another law of telephony physics is called the Busy Call Completions per Hour or BCC or BCCH. For whatever reason in the digital age of telephony, calls can’t be made and placed at a single time. During small (but large to us) earthquakes, lines will jam up and what happens is callers will hear dead air. You can make DTMF tones, but that’s all you’ll get till the ESS switching gets restored. Is this necessary? I’m not a PBX or ESS engineer, but for whatever reason, most landlines can handle 100,000 completed calls per hour. (This is very important because what if someone is calling their friend just to “check in” and you were the last completed call and another person is dying of a heart attack and they are the one that will hear empty DTMF tones to 9-1-1.) This is why it gets repeated during a natural disaster or an unusual event (like an earthquake on the East Coast) to not use the phone unless its to call someone like a 9-1-1. That’s also to say do not call 9-1-1 for non emergencies during major events. In the February 1993 bombing of the World Trade Center, before 3-1-1 was ever a common number, the New York City Fire and Police departments were indicated with calls, despite the New York Telephone network working without massive outages like someone not blowing the BCC limit, it was their 9-1-1 system that was blowing the BCC limit. (of course I am talking about the rest of the Manhattan island and the other four boroughs – it was given that lower Manhattan had little dial tone.)
In closing, it would not be unusual for an event like today to have calls drop and get announcements from the provider that “all circuits are busy.”
Another problem that people are not familiar is this “triple play” technology is no different than having SIP, over the top or broadband Internet. This is called DOCSIS, the protocol behind “triple play”. Basically all your services run on an IP network, but you don’t see it that way is because by design it works like a traditional cable box, or your phone works like an ordinary wireline service or your Internet is a high speed Ethernet connection.
The dirty little secret? All of this runs on an IP network. Not like your civilian/corporate network, but instead of having a Cat 5 cable to plug into a wall like Roku, it terminates via coax, and basically the coax goes to the “last mile” boxes in your neighborhood and through fiber goes to your headend. But it’s at the headend the “magic” of SIP like VOIP, and OTT like video is being processed. In short, your headend and your devices are basically the broadcast equivalent of the Internet of Things or IOT. I am not indicating that Comcast had a breach to their video network, but the chances of a hack are typically unlikely because a lot of this is done through secure networks and the least amount of hardware possible. Again was this a hack job or what, is not confirmed, and yes a possibility of this happening is there.
If this was a hack job, then there is more to be concerned about “the rest of us” and our civilan (err consumer) grade IOT devices like Smart TVs and wearable technologies.
Despite the outcomes for Comcast, it’s important to put this unusual outage of Cable TV services and their call center outages into proper context and perspective. There will be Monday Morning Quarterback hangover in Philly tomorrow and I would suspect there will be damage control. But please wether you love Comcast or hate them, today isn’t the day to complain. Coming from someone who understands the telecom technologies.