Routine Fiber Maintaince in Port Orchard this morning cost me $100K

This morning, a routine maintainence event went south in a bad way. The McCormick Trails/Village area is a high density housing area with a large presence of KPUD fiber. KPUD was improving one of their fiber cases to make room for the next phases of expansion in the area. Yawn, status quo. Homes were expected to go offline at 1AM (check, 1:01AM they went offine), homes were due back online by 5:00AM. At 5:45 our phones started blowing up. The work went long; KPUD had difficulty restoring service. On top of that, Murphy dropped power at the Bremerton airport upstream of the neigbhorhood. It was a double fault.

We use NoaNet (KPUD is an investor/partial owner of NoaNet) as our backup internet source. We transitioned to Lumen, with a 100gb feed as our primary over a year ago. Our NoaNet backup comes over the KPUD fiber network from Bremerton. We have been nervous about having a fault that takes down our NoaNet backup at the same time there is a partial network disruption in Kitsap. To avoid this, we have been negotiating with Cogent (another Tier-1 global network operator) to get a 10gb backup to supplement our NoaNet connection.

The outage this morning sealed the deal. The ink is dry. We just signed a $100k three year contract to get 10gb service as a backup we hope to never use.

That is $100K straight off our bottom line that could replace the 2005 Dodge minivan you may see us pull up to your house in. Rest assured, we are more interested in amazing internet than flashy rides. And no, I won’t sell you my minivan, don’t ask.