October 12th, 2009
In the aftermath of the T-Mobile Chernobyl
T-Mobile’s Sidekick/Danger Inc. meltdown has far-reaching implications for wireless carrier data integrity responsibilities way beyond any immediate remediation the company may need to offer its customers.
Yesterday, T-Mobile notified all of its Sidekick device customers that a failure which occurred on Microsoft’s Danger, Inc. subsidiary’s Hitachi-based storage systems caused a catastrophic data loss at a central data center which affected approximately 800,000 of their subscribers worldwide.
This data loss included all contacts, digital photos, calendar entries and to-do lists that were stored by each customer because much of the device’s core functionality was entirely dependent on online, cloud-based services.
As a consolation, T-Mobile has generously offered its customers either an early termination escape from their contracts at zero penalty, or a $20 credit on a T-Mobile G1 android phone.
Really, T-Mobile? REALLY? ARE YOU FREAKING KIDDING ME?
Click on the “Read the rest of this entry” link below.
Let’s get into the immediate consequences, shall we? If T-Mobile doesn’t want to lose a lot more than just 800,000 customers overnight, they’d better do a hell of a lot more for their subscribers than offer a zero-penalty early escape or a measly $20 in credits towards a new device.
If I were T-Mobile, I’d offer each and every customer a credit that would be the equivalent in value to the contract early termination fee that would normally be collected by T-Mobile, which could be then applied to an inexpensive and reconditioned and fully-warrantied smartphone of the customer’s choice or a substantially discounted myTouch 3G phone at new customer pricing with the remainder of that credit applied to future data service fees. Nothing, and I mean absolutely nothing, should have to come out of the customer’s pocket due to this screw up.
Yes, I realize that such credits would amount to a lot of money, especially if you consider that early termination on a year of remaining service is probably around $125-$150. At $125.00 per customer, that’s $100,000,000. But the impact of losing the ongoing revenue stream of 800,000 customers as well as being party to multiple class-action lawsuits which could result in losses in the several billions of dollars if it doesn’t provide immediate customer satisfaction. A hundred million dollars or even two hundred million is a drop in the bucket if you consider what the worst case scenario for T-Mobile as well as for Microsoft is.
[EDIT: The following was posted on T-Mobile's Sidekick bulletin board at 5:15PM PDT: In the event certain customers have experienced a significant and permanent loss of personal content, T-Mobile will be sending these customers a $100 customer appreciation card. This will be in addition to the free month of data service that already went to Sidekick data customers. This card can be used towards T-Mobile products and services, or a customer’s T-Mobile bill. For those who fall into this category, details will be sent out in the next 14 days – there is no action needed on the part of these customers. We however remain hopeful that for the majority of our customers, personal content can be recovered.]
I for one would like to know what the criterion for “significant and permanent loss” is. EVERY Sidekick customer should get that $100 credit, at a bare minimum, and Microsoft should have to ante up on a large portion of those credits as the responsible party.
But what does this catastrophe mean for wireless carriers that have services that store customer data in private clouds in the future? Well, I think it means that they all had better get their infrastructure maturity where it needs to be and their business continuity and resiliency ducks in a row so that this never, ever happens again. The fact that Danger Inc. was able to have such a catastrophic failure and complete data loss of this magnitude means that the infrastructure was poorly managed and poor data protection methods were in place.
As much as others might want to point fingers at the hardware that failed, I can’t place the blame for this on Danger’s storage vendor, Hitachi, or its server vendor, Sun Microsystems. Hardware fails, and it doesn’t matter what enterprise storage or systems vendor you’re dealing with, whether it’s Hitachi, EMC, HP, Sun or IBM.
The fact of the matter is this type of failure only occurs because Danger went cheap on data protection and their IT Management controls sucked. I don’t even have to know the details of the failure to know that this has to be true. Shame on Microsoft for not ensuring that the infrastructure wasn’t brought up to spec to the rest of the company’s standards after they acquired it, and shame on T-Mobile for not ensuring that their customers’ data was safe.
If enough arrays were in place, not only would the company have had local and off-site disk-to-disk storage replication as a best practice, but they’d have snapshot and disk-to-tape capability so that even if you had a freak data corruption error that would destroy all of the replicas (a very rare occurrence indeed) that you’d have the ability to go back a day, a week, or whatever their last snapshot was. This is what real enterprises — banks, insurance companies and other large institutional entities have to do for their customers, because in many cases government agencies insist on it.
Do we now have to treat wireless service provider customer cloud-stored data with the same level of care as large financial institutions do? Does T-Mobile need to “suck it up” and provide a significant credit to each customer in lieu of their failure? Talk Back and Let Me Know.
Jason Perlow is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. See his full profile and disclosure of his industry affiliations.
Subscribe to Tech Broiler via Email alerts or RSS.
| 10/14/09
