May 30th, 2007
Massive network outage for AT&T-SBC
1:00 AM, I’m working late night here in Silicon Valley and the Internet Naming service has gone dark which means I can’t get to the Internet. Not a single DNS server on the AT&T-SBC network functions which tells me it’s a bigger issue than just a DNS server problem. I even fired up my own test Windows Server 2003 virtual machine that has a full fledged DNS server installed and even that wouldn’t function. That means access to the Root DNS servers from the AT&T SBC network is being blocked.
2:30 AM, I finally get through to first tier tech support and they tell me that DNS is down for all of California. That of course doesn’t mean that DNS is down, just that none of California can get to main AT&T SBC DNS servers. Since my virtual server can’t even resolve names, that means access to most of the Internet and the Root DNS servers on the Internet are severed.
2:50 AM, I get forwarded to second tier tech support and the tech explains to me that the Pleasanton CA POP (Point of Presence) is undergoing “scheduled maintenance” from 12:00 AM till 6:00 AM central time and that there are outages throughout AT&T’s network as well as other ISPs that use AT&T’s network. If your network went down and you use AT&T as a carrier for part or all of your network, now you know why and who to “thank”.
If it’s scheduled, I guess AT&T saw fit to schedule 6 hours of outages without notifying its customers. But according to the technician, this isn’t actually an “outage”. But if this is scheduled, the network traffic could have been rerouted ahead of time but I guess AT&T figures we don’t need the service and nor do we even need to be notified. It sure looks like an outage to me since I had to do an hour or more of troubleshooting to track down the problem and waste all that time and spend another hour or more on the phone with tech support. ASI (Advanced Solutions Inc) which is a subsidiary of ATT is in charge of that network and they brought the entire network down by working on the routing infrastructure with no backup route in place. Those of us that are unfortunate to not have redundant BGP (Border Gate Protocol) paths are just out of luck but hey, 6 hours of downtime isn’t really an “outage”.
We can all “thank” AT&T SBC and ASI for having the customer in mind. We’ve been declared a “maintenance” zone which really isn’t an “outage”. We can also “thank” them for the fact that they felt that there was no need to alert and alarm anyone.
The only reason I can post this now is because my virtual server was able to figure out the IP address of CNET’s servers and I hacked my hosts file to be able to manually get here without the help of DNS.
George Ou is Technical Director of ZDNet. See his full profile and disclosure of his industry affiliations.
| 05/30/07
