Monday, September 22, 2008

One Week Later - A Question of Reliability

We are a week after the flood and everything is back to normal at our schools. The same can't be said for all of our families, some of which are still homeless or have been greatly impacted by the local flooding.

Last week was an interesting test for us. Over the past year we had fiber optics installed for our data network and we have been transitioning off of our T1 lines. Currently some of our systems are on fiber, some are on T1, and some ride between them.

Last week's flood put an important box of equipment (somewhere between us and AT&T) underwater and behind police barricades. We lost part of our network for 48 hours. Here is what I learned.

1. The more dependable your network becomes the more people rely on it.

  • 2 years ago network outages were common here. People counted on the network going down at times. Over the past year we have been down for no more than 10 minutes at a time and but one instance was a planned occurrence (that one was AT&T's fault). Nobody counts on it going down anymore and most everyone is unhappy when it does.
2. Give people a tool to communicate with and they will use it. Take it away and you cripple them.
  • Last year we added Blackberries to our communication structure. Part of their infrastructure was using the T1 that went out. Without the Blackberries providing email we went through a communication interruption. It was difficult to quickly get a hold of anyone, plus many people didn't have each other's phone numbers because they always email each other. We have to fix that one
3. Break the network hear from everyone - fix it and nobody calls.
  • Ok this may be whining, but here goes anyway. How many people called to thank us when we hacked stuff together to get the blackberries working 24 hours into the outage - nobody. How many people were outraged that it didn't work right and they were inconvinienced - lots. Yeah, I am whining.
Lesson learned - our disater management needs some work.