Fun with Phone Lines
We provide “Land line replace service” for folks switching to fiber. We don’t know why people still want landlines, as most Americans over the age of 15 have a mobile device. However, there are many use cases we still support. The most common need is a simple phone a child can use for 911 or an older customer who is more comfortable with a handset than a cellular device. We built our phone service on a very good platform (at the time, in 2013) and have stayed with them as we have grown. Once we passed the 400 phone lines mark we began to run into rare occasions where our phone servers and their systems would decide they don’t like each other. All incoming calls for customers would return “busy” until I manually stopped, let the dust settle for three minutes, and restarted our server. This allowed both sides to “take a time out and cool down”. We reached a head about a month ago and I told the upstream “This cannot happen again, we pay you too much money monthly to suffer this uncertainty.” They agreed! On 4/3, starting at 6:30AM, we worked with their network engineers to migrate 548 phone lines and 23 “PBX” phone servers to a new communication platform. About halfway into it, we ran into a snag, and I commented “I wish I knew more”; their engineer replied “This is one of the easiest calls this week, you know more than most customers”. This was from a telco network engineer who works at a company that handles over 300 billion minutes of phone calls per year that covers 93% of America.
The old platform had a “username/password” that we had to send every six minutes to keep things working. The new system uses a dedicated internet address reserved for us. Both ends are “reserved” and both parties control the IP numbers/blocks. Think of it as a “VPN” without the P, it is a dedicated point to point connection where both parties trust the other is allowed to speak. There still is all sorts of “per call authentication”, but the server to server is assumed trusted. We can do this as we cryptographically announce our IP address to all the tier-1 routers in the world (Lumen/Cogent/Zayo/Hurricane Electric/Comcast/etc). There isn’t any “man in the middle” who can insert themselves to mess with our traffic.
I have been dreading doing this changeover for a few years. After the last big outage (where I was unavailable for a few hours due to family reasons) I had one of those “jeez Steve, what if you get hit by a bus” moments and realized we have to make an immediate, measureable, improvement in our service.
I am happy to report that we had only one snafu out of all that shuffling affecting only one phone number serving one company with three handsets.
This is a long winded note to explain, “phones are really, really hard to do well”, we take them seriously and strive for perfection.