A tree trimming to remember
Morning of September 19th, 8:45AM, a tree trimmer did their best effort to destroy day one of my 30th wedding anniversary trip. The chainsaw artist decided to cut through KPUD’s fiber backbone. This took down our new in county 40gb link to our core router and all the commercial services on Bainbridge, including Police, City Hall, Fire, Cell Towers, Ferry Terminal, School District, the Winslow shops we serve, and about 300 homes, including our own. About a month ago I made a live change to the new 100gb core switch during daytime hours which caused a crash. I haven’t manned up to redo the configuration to enable auto failover. This particular link of fiber hasn’t had any damage in the fourteen years we have been doing this. What’s the chance something will go wrong on just five miles of fiber in the county (out of hundreds) while I am gone for a week? Murphy read my mind and responded in full force. Because we have in county redundancy, the next generation of engineers were able to reconfigure promptly once we understood what the problem was. It took a while to figure out what was wrong as the break took out so much that both KPUD and Net253 thought the entire north Bainbridge facility might be offline.
KPUD rallied and we had rapid response from MLS, one of the telecommunications line crews who happened to be on island today for a construction project we were doing with them. Needless to say, our project took back burner. KPUD will be working until all services are restored, estimated to be done around midnight. This is hundreds of fiber patches (times two for both ends of the new fiber segment) This one stunk. It could have been better if we had autofailover working. I will make that a priority on return. I had to take this 30th anniversary trip if I want a 31st (I was informed of this). So hopefully you will cut me a little slack. We do have the physical redundancy, so we were actually able to recover before the lines are repaired for most of the county. Net253 is the only provider with that level of redundancy in the county. Our planning and investments paid off.
Tree trimmer gone wild. Really looks like they nailed the fiber and then said “oops” and gave up and moved to the next section.
Aerial crew hanging almost 1000 feet of new fiber to repair the break and install storage (so that future breaks have material to rapidly fix)