Network cleanup, went south fast

We have a KPUD DIA (Business Internet) link with KPUD since we moved into 750 Ericksen. Every month I pay $20 for a block of five addresses on the internet but only use one.

We tried to roll back to just one address (after all, I have paid $1,920 for nothing at this point). I fat fingered one character.

As a result, all our customers who have business circuits and phone service from us were suddenly unable to use their phones.

After about thirty minutes, we experienced the classic “bell curve of frustration”. It got to the point where I couldn’t think because the phone was ringing constantly.

We are working on a 100gbit upgrade for our county link and we have been super careful to do that during very off hours (3-5am on saturday mornings). This other change is brain dead simple; we tested it and it seemed to work at first blush.

It is always the easy stuff that trips you up!

Anyways, problem fixed, 800 people no longer mad at me, now to get some late lunch.