Everybody was scrambling this morning, as the servers were down. It appeared to be a routing issue, upstream from us. We had Marisa, Cory, Sean, John, Shari O., and myself on the meeting. We also had Bryan join for a bit. Steve was texting back and forth, and Wayne joined and gave us more information as well.

The whole tech support team was answering emails, text message, and trying to help pass on the information that we had. Great team effort. The sad thing was, we couldn't really control all of the variables. The routing problem was even above the data center or server farm level. We fix code and can tweak a few things, but if the main highway is broken, I don't care how good your vehicle is, it just won't go.

A few small lessons learned... We were trying to send out an email blast, but most of our data was on the primary adilas.biz servers and we couldn't get to it. We did have some older exports of our client list, but it was a few months old. We also updated some files, through FTP, but the caches weren't being updated. Interesting.

As the time progresses, we kept fielding calls, texts, emails, etc. Wayne and John were doing a great job bouncing around and checking all kinds of different maps, up/down internet sites, and even bouncing to different countries to try to access our sites from different IP addresses and access points. Pretty crazy.