February 20th, 2008

Remember this, BlackBerry: when it comes to customer communications, it's NOT all about the carriers

Posted by Russell Shaw @ 9:16 pm

Categories: Carriers, News, Outages

Tags: RIM BlackBerry, Outage, Carrier, Handhelds, Hardware, Russell Shaw

Once again, there were scattered BlackBerry service outages yesterday. This time, it was maintenance on the BlackBerry Internet Service side of the business. The one that forwards your other email account messages to your BlackBerry.

Unfortunately, yesterday proved that when it comes to letting customers know what the you-know-what is going on, BlackBerry’s notification processes are extremely flawed.

These outage notification processes seem to entirely consist of:

Email alerts directly from RIM to key commercial accounts.

There doesn’t seem to be a system in place for letting smaller accounts, and individuals who access BlackBerry services via their cell carrier from knowing what is going on.

No “we have reports of scattered BlackBerry outages” notifications on the BlackBerry and RIM sites. There also doesn’t seem to be a regular structure in place for letting carriers know of these disruptions in a manner that would enable these carriers to post outage-related notifications and updates on their own sites.

These have been flaws ever since I can remember.

Plainly, these flaws call not only for engineering new business processes that let all BlackBerry customers know about outages.

If it means taking a few carriers by the scruff of the neck and getting them in the customer alert loop, then so be it.

I still have my doubts this will be done right away. Even if the will to do so was there, BlackBerry is so subservient to their carriers I doubt they lack the will or the vision to architect subscriber alert systems such as the type I am proposing here.

February 20th, 2008

Numerous BlackBerry BIS service outages reported

Posted by Russell Shaw @ 9:30 am

Categories: News, Outages

Tags: RIM BlackBerry, Outage, Handhelds, Hardware, Russell Shaw

Numerous posters on BlackBerry board Pinstack are reporting some BlackBerry BIS service outages.

“I love my bb, but this is getting ridiculous,” one poster writes.

I beg to agree.

Update: Reports are coming in pointing to issues with “scheduled maintenance,” but generally attesting to network back up around 4 p.m. ET/1 p.m. PT. 

February 19th, 2008

Analyst: BlackBerry service glitches ignite "concerns," but no harm done (yet)

Posted by Russell Shaw @ 8:12 am

Categories: Corporate Stuff, News, Outages

Tags: Research In Motion Ltd., Network Operations Center, RIM BlackBerry, Sales Strategy, E-mail, Manufacturing, Handhelds, Sales, Online Communications, Hardware

I’ve just been sent a copy of BlackBerry-maker Research In Motion stock analyst Mike Abramsky’s brand new, quite detailed analysis on RIM.

There’s plenty to chew on in this RBC Capital Markets report.

The first thing I wanted to know is Abramsky’s take on what effect, if any, the Feb.11 BlackBerry service outage will have on RIM’s business.

Summary: the incident reignites “concerns,” but wont effect sales of new BlackBerrys.

On Feb 11, RIM suffered a service disruption, leaving 8M BlackBerry users in the Americas without service for 3 hours, due to a problem with a system upgrade intended to increase capacity. The outage offers ST headline risk and reignites concerns over dependencies on RIM’s service, but is unlikely in our view to affect sales momentum (RIM’s last disruption was 10 months ago).

Abramsky then gives us some info on the Network Operations Center (NOC) where the outage occurred.

We est. RIM’s centralized NOC architecture now handles est. up to 1-2B emails per day (75-150 emails/day/user x 12M subs) or equivalent to 10%+ of global phone traffic, growing at 72% CAGR. The NOC also provides RIM with its competitive advantages — push email, security, bandwidth efficiency, etc. and offloads overhead from internal email servers, which are often less reliable.

Hmm, quite a busy place.  Apparently too busy to institute robust redundancies?

February 13th, 2008

NOC-NOC, who's there? Or, "I've seen the thumbwheel and the damage done"

Posted by Russell Shaw @ 11:10 am

Categories: Corporate Stuff, Outages, Security

Tags: Research In Motion Ltd., RIM BlackBerry, Outage, Tom, Manufacturing, Russell Shaw

NOC= (Blackberry-maker Research In Motion’s Network Operations Center in Waterloo, Ont.)

As to the latter portion of this post’s title, NOT an original.

Readers of a certain vintage may recognize that saying as an adaptation from an old Neil Young song entitled “I’ve Seen The Needle and the Damage Done.”

My colleague Tom Krazit gives the phrase a fresh meaning as he puts up a post containing observations about BlackBerry-maker Research In Motion’s Tuesday service outage, as well as continued practice of running pretty much all BlackBerry email and data traffic through its Network Operations Center.

Tom writes in part:

The Wall Street Journal reported Tuesday that expansion efforts at RIM’s NOC may have been to blame for the outage. The problem isn’t that the servers are in Canada; they could be anywhere. It’s just that everything has to go through the one location. In theory, as long as you have enough redundant backup systems and plans, that shouldn’t be a problem. But every now and then, it is.

Frank Gilman, the chief technology officer for Los Angeles law firm Allen Matkins, was forced to deal with the outage Monday afternoon. “What surprised me was the apparent lack of a solid business continuity plan on RIM’s part to ensure reasonable connectivity,” he said via e-mail, of course. “A company that is marketing devices that increase the mobility of professionals should have systems and contingencies in effect to avoid an outage of that size and duration.”

This can lead to problems.

As Tom points out, and I so totally agree, not having multi-site redundancy for when this sheet hits the fan kind of leaves enterprises who have put full faith in BlackBerry’s “five nines” Service Level Agreements in a real pickle.

That’s so not OK.

If you are an enterprise BlackBerry user, I’d love to read your thoughts on this. Would you rather that RIM offer more redundancy, including perhaps, addl NOC(s).

February 12th, 2008

BlackBerry: outage probably caused by data routing system upgrade glitch

Posted by Russell Shaw @ 3:44 pm

Categories: News, Outages

Tags: Upgrade, RIM BlackBerry, Outage, Manufacturing, Handhelds, Hardware, Russell Shaw

BlackBerry-maker Research In Motion has just said an early investigation is pointing to a problem with an upgrade of a data routing system.

The upgrade was part of an ongoing effort to expand capacity for long-term growth, it noted.

More as we learn more.

February 12th, 2008

What will-and won't-work during the next BlackBerry service disruption

Posted by Russell Shaw @ 8:18 am

Categories: Outages, Tips & Tricks

Tags: Network, RIM BlackBerry, SMS, Outage, Hayden, PIN, Text Messaging/SMS/MMS, Handhelds, Manufacturing, Cellular Phones

Hayden, who runs the superb Pinstack BlackBerry forum, offers a post with some practical advice the next time it appears that your BlackBerry service has gone down.

The first thing you should do is run a test to determine whether it is just your BlackBerry, your network- or as we saw yesterday- issues that are totally more widespread.

Specifically, Hayden counsels that you should:

Run a BlackBerry Network dianostics test. This will confirm if you are connected to BlackBerry servers or not. Go to: Options > Mobile Network > (Press menu button) > Diagnostics Test > Run.

OK, so we have an outage. Besides not panicking, what do we do now?

Hayden writes:

Don’t always expect PIN to PIN messages to work - During outages don’t expect PIN to PIN messages to work sometimes that service will also be down. Check the pin-to-pin status in your diagnostics report. However, you can still “send” a PIN message but it wont be delivered till service is restored.

Use SMS text messaging - Make urgent contact using SMS text messages. These are sent over your Carrier’s Network by default unless you’ve changed that setting in Options > SMS text > Network to send over.

You’ve been Temporarily downgraded! - Remember during an outage your BlackBerry will function similar to a regular low-end cell phone. By this we mean No PUSH email, no web browsing, no PIN messaging and no IM (Instant Messaging) services.

Repeat the Diagnostics Test when service is restored - If the test fails go to Manage Connections and turn off your Mobile Network option, then check box to turn back on. Still fails? Hold in the power button till the device powers down, then press the power button again to power up. If the service fails and you are 100% certain it’s been restored then press and hold power button to shut down device, then remove and replace the battery for a hard reset.

Hayden then offers this Bonus tip:

Reply to pending emails via PC if possible. Yes we know you are itching to get those thumbs typing but with any outage there will be a flood of messages and data being syncronized once service is restored. Help ease RIM’s initial recovery-load by avoiding the use of BB email, browsing, etc for at least the first 30mins unless of course you are away from your desk.

February 11th, 2008

Source tells me: IP address glitches likely a factor in today's BlackBerry outages

Posted by Russell Shaw @ 6:25 pm

Categories: News, Outages

Tags: IP, IP Address, RIM BlackBerry, Outage, Network Technology, Networking, Russell Shaw

A representative of BlackBerry traffic analysis solutions provider Zenprise softwarei just got back to me with the scoop of some diagnostics that are pointing to at least one problem likely at the root of  today’s bad BlackBerry North American services outage:

Basically there are two paths (IP addresses) in North America to connect to the RIM network.  According to diagnostic tests run by Zenprise software, one IP address was refusing connections to come through causing enterprise users to be impacted.  A few points to note:

* Any users on the working IP address experienced little to no service interruption

* Organizations that reported intermittent email activity are the result of switching between the two IP addresses.

Still working on learning more. But the problem does seem to be dissipating as I report this.

February 11th, 2008

RIM: here's what's going on with the service outage (and it ain't good)

Posted by Russell Shaw @ 3:06 pm

Categories: News, Outages

Tags: Research In Motion Ltd., RIM BlackBerry, Outage, Handhelds, Hardware, Russell Shaw

This from BlackBerry-maker Research In Motion:

BlackBerry subscribers may be unable to send or receive messages. Subscribers may also be unable to register their device, roam in another location, or use other services such as Internet browsing.

BlackBerry Internet Service subscribers may be unable to use the BlackBerry Internet Service web site or perform activities such as creating new accounts, accessing their Internet mailbox, integrating third-party email accounts, or viewing email attachments.

Devices may not receive new service books. BlackBerry Connect and BlackBerry-enabled devices that require a new PIN may be unable to receive the PIN.

BlackBerry Enterprise Servers may be unable to connect to the BlackBerry Infrastructure.

Wireless service providers and device resellers may be unable to use BlackBerry administration web sites or perform activities such as creating subscriber accounts or provisioning services for subscribers.

You know, starting to think this stuff is happening a bit too often.

Do you agree?

February 11th, 2008

RIM: "Critical Severity" BlackBerry outage

Posted by Russell Shaw @ 2:22 pm

Categories: News, Outages

Tags: Research In Motion Ltd., RIM BlackBerry, Outage, E-mail, Manufacturing, Handhelds, Online Communications, Hardware, Russell Shaw

Sounds serious, and it is.

As an AT&T Mobility BlackBerry subscribver, I first started to pick up something wrong when I sent some photos of me swimming in a spa pool yesterday to some of my other email addys.

They aren’t getting thru.

But it’s not just me. Reuters just moved this:

E-mail service of Research In Motion’s BlackBerry smartphones experienced a “critical severity outage” on Monday, the company told clients in an e-mail.

“This is an emergency notification regarding the current BlackBerry Infrastructure outage,” stated an e-mail from RIM support account manager Bryan Simpson. The e-mail said the outage affected enterprise clients and “users of the Americas network.”

BlackBerry forum Pinstack has more.

January 31st, 2008

AT&T BlackBerry outages reported

Posted by Russell Shaw @ 4:48 pm

Categories: Outages

Tags: AT&T Corp., EDGE, Outage, 3G, Cellular Phones, Consumer Electronics, Personal Technology, Russell Shaw

Reports have been pouring in for the last several hours.

AT&T BlackBerry service, which run on AT&T’s not so edgy EDGE network, are really having a rough patch with connectivity today.

As BlackBerry board Pinstack reports via Engadget:

We’re getting reports from all over that AT&T customers are having trouble with 3G and EDGE data. A majority of the complaints are coming from the Midwest / Chicago areas, where things seem to be down entirely, but we’ve also heard from folks as far away as Florida. Team Engadget hasn’t hit any snags in New York or Vegas where we’re currently situated, but let us know if you’re having troubles wherever you might be — and what color crayon to use for this fancy “threat level” printout we’ve got pinned to the wall.

Is your device, and connectivity, affected?