RIM apologises to South Africa

Research in Motion has launched a massive public relations exercise to repair the damage caused by BlackBerry’s three-day outage and the silence of its executives. While founder Mike Laziridis took to YouTube, RIM’s head of regional marketing answered questions in a teleconference with South African media – including Gadget.

On Day three of the biggest outage in the company’s 12-year history, Research in Motion finally swung into action with a massive public relations exercise.

In an effort to repair the damage caused by the outage and the silence of its executives, founder and joint-CEO Mike Laziridis took to YouTube: RIM’s United Kingdom managing director Steven Bates addressed the media outside his headquarters: and Patrick Spence, Managing Director Global Sales and Regional Marketing at RIM, answered questions in a teleconference with South African media.

Gadget was there with questions our readers have been asking, but the first order of business was a heartfelt apology.

Patrick Spence, Managing Director Global Sales and Regional Marketing at RIM

‚The first thing I would say is that we’re extremely sorry for this incident and completely understand the frustration of our customers and partners in experiencing messaging and browser delays,‚ Spence introduced the press conference.

‚The good news is that, since 6am British Summer Time all services have been up with sign of continual improvements and traffic is flowing as expected. There may still be some backlogs and delays in terms of messaging.‚

He explained that the failure of a core network switch ‚ which manages the flow of data through the system – had snowballed after the backup also failed.

‚All our network switches have multiple redundancies. Unfortunately in this case the failover did not function as we expected it to, despite the fact that we regularly test our systems.

‚As a result, and because we are moving 22 petabytes (22 000 terrabytes or 22 million gigabytes) of data through our system every month, a large backlog of messages was generated and it has taken us an extended period of time to restore the system.

‚We have built a whole bunch of layers and redundancies into our systems, but in this case the backups failed. Our first order of priority is to ensure that all systems are back up and operational, and that’s what the teams have been working on.‚

Restoring public confidence

Spence was frank about the bigger task lying ahead: restoring public confidence.

‚As we turn to the second order of business, as we have everything back up, and we are confident we are completely back to normal, we will be focusing on three things:

‚Firstly, a full assessment of processes, systems and architecture to make sure we avoid this kind of issue in future.

‚Secondly, how do we improve communications around these kinds of issues? We did start to communicate issues 10 minutes after it happened on Monday but, clearly, based on feedback, we haven’t done a good job.

‚Thirdly, How do we make up to our customers and what do we do on that front?‚

This immediately raised the question of compensation for customers. Spence would not be drawn on this approach, but did not rule it out.

‚We are very focused on getting everything to where we need to and that is a question to which we will turn our attention tomorrow. We will see what we have learned from a communications perspective on a number of fronts.‚

On the question of whether the system had been overloaded, he insisted that it was designed to withstand current loads, and would be scaled up as the load increased.

‚We carefully watch that and build ahead of it . The challenge is more one of dealing with 22 petabytes of data and a rare switch failure like this. The issue is not how the system handles data, but how it gets back to handling that amount of data in the way it normally does.

‚We do tons and tons of testing for reliability and tolerance We keep our network running well ahead of the trends we see. There is not a capacity issue. That’s why, yesterday, some people were functioning properly and coming through, and gradually traffic patterns wil be back to normal.‚

Spence reiterated that not all of BlackBerry’s 70-million users were affected.

‚It has been intermittent. It hit some but not others. The reality is we have to serve all customers and we are clearly sorry that even one customer was affected. We’ve been so focused on making sure we get service restored, we haven’t had a chance to address this, but as a second order of business, we will be looking at what we have to do to win back the trust of our customers and partners.‚

Spence insisted that all services were running at normal levels. However, ‚there are still some messages flowing through the system, so there is some backlog for some users. It is looking normal from a traffic point of view. I want to make sure we get through the entire day with the kind of reliability we expect before we turn to other things.‚

The communications outage

Among the issues that will be addressed as part of the ‚second order of business‚ , he said, was ‚how we avoid this in future‚ .

‚We are going to dig very deep and leave no stone unturned to find out what happened and why. I’ve been at RIM for over 13 years, and have never encountered something like this.‚

The critical issue for RIM, aside from the outage itself, was the communications outage. That would be a critical element of the ‚second order of business‚ :

‚We have a notification system for enterprise customers and partners, so operators and enterprise customers would have got notifications in terms of the incident. Looking at it in hindsight, pretty much every one was ‚all hands on deck’ to resolve the outage, and we have to balance that with how much attention we pay to communications. We did have plan, but it clearly did not go broad enough. We have to learn from this and do better in future.‚

From a brand damage point of view, Spence believed it was ‚one where we’ve got to get the service back to level of reliability everyone expects, and have to prove it’s a rare incident, but we have got to prove it and earn trust back, starting today, in terms of delivering messages when we say we will. Hopefully the equity we built up over past 12 years will carry us through this. Customers are the most important thing at end of day. We are now trying to share as much as we can.‚

Spence had been with co-CEO Jim Balsillie in Dubai on Monday when the crisis began. He told the press conference that his boss had been extremely concerned, and was immediately in touch with senior staff, who provided him with regular updates.

‚We were meeting with many media, partners and government officials in Dubai at the time. It was immediately and all-hands-on-deck approach. We were feeling the affects immediately, and Mike (Laziridis) has been on top of it since then.‚

Asked whether RIM would consider situating a data centre in South Africa to handle local traffic, Spence said it could not be ruled out.

‚We do have data centres spread throughout the world. We don’t reveal the exact locations due to potential security concerns, but certainly, as we go through the causes and how to avoid this situation, we’re not taking anything off the table. If something like that would help, it’s something that would be on the table and considered. There is nothing we wouldn’t do to provide customers with reliable service.‚

He said that BlackBerry users who still experienced problems should contact their own service providers’ support centres, but could also visit the BlackBerry Service Update page (http://www.blackberry.com/serviceupdate), which will shortly include advice about resetting devices. The simplest approach is to remove the battery as a ‚hard reboot‚ of the device. That tends to resolve any lingering connectivity issues.

Laziridis speaks

Earlier, at about 8am Eastern Time (2pm South African time), Mike Laziridis, founder and joint-CEO of RIM, issued the following video statement on BlackBerry’s official YouTube channel (view it here: http://www.youtube.com/blackberry).

Mike Laziridis, founder and joint-CEO of RIM

Since launching BlackBerry in 1999, it’s been my goal to provide reliable, real-time communications around the world. We did not deliver on that goal this week, not even close. I apologize for the service outages this week. We’ve let many of you down. But let me assure you that we’re working round the clock to fix this.

You expect better from us, and I expect better from us. It’s too soon to say that this issue is fully resolved, but let me give you more detail about what’s happening. We are now approaching normal BlackBerry service levels in EME India and Africa. We continue to monitor the system very closely.

We are working very hard to continue to stabilise the system and we are seeing steady improvements. We expect to see continued progress and possibly some instability as the system comes back to normal service levels everywhere.

We know you want to hear more from us and we’re working to update you more frequently though our web sites and social media channels as we gather more information.

I’d like to give you an estimated time of full recovery around the world but I cannot do this, most certainly at this time.

For those of you affected, I know this is very frustrating. We’re doing everything in our power to restore regular service levels. And we’re working tirelessly to restore your trust in us. We’ll update you again soon.

email this to a friendnttnntt printer friendly version

“,”body-href”:””}]