April 24th, 2009

Twelve large IT failures to remember

Posted by Michael Krigsman @ 3:14 pm

Categories: News roundup, Project failures, Uncategorized

Tags:

IT failures are so common that it’s hard to pick out unique examples worth looking at in closer detail. I usually examine failures that are large, serve as good teaching examples, or demonstrate severe organizational hubris.

Although I question just how useful such lists actually are, their popularity can’t be denied. In that spirit, here’s a top-ten (well, twelve in this case) list from CIO Insight magazine. The related links refer to more in-depth blog posts I’ve written.

1. IBM’s Team for Texas Consolidation. The State of Texas partnered with IBM to consolidate data centers across the state. Service levels have dropped so low that many agencies gave IBM failing ratings and complained it was taking too long to perform routine tasks. What went wrong: Poor project requirements and a lowest-bidder vendor pick. RELATED: Texas gov. intercedes in failing IBM project
2. US 2010 Census Handheld Rollout. The ambitious plan to arm US Census Bureau employees with handheld computers to compile and transmit 2010 census information to headquarters was mostly scrapped after two years of work. What went wrong: Scope creep. RELATED: Billion-dollar IT failure at Census Bureau

Read the rest of this entry »

February 27th, 2009

IT failures roundup: Banks around the world!

Posted by Michael Krigsman @ 5:26 am

Categories: Availability and reliability, End-user impact, News roundup

Tags: Software, Bank, Information Technology, Debit Card, Online Banking, UBS AG, Australian IT, Banking, Financial Services, Personal Finance

Failures seem to come in waves, and today it’s banking. Think your bank’s systems are perfect? Don’t be so sure.

“System error” at UBS Japan causes erroneous $31 billion trade. From Finextra:

Swiss bank UBS has confirmed that a “system error” was responsible for the entry of an erroneous trade for three trillion yen at the Tokyo Stock Exchange on Tuesday.

UBS told the Bloomberg newswire that its Japanese unit mistakenly ordered $30.9 billion of Capcom Co. convertible bonds, 100,000 times more than it intended, because of an internal system error.

Allied Irish Bank incorrectly pulls cash from customer accounts. Again, Finextra has the story:

Read the rest of this entry »

February 11th, 2009

IT failures roundup: software bugs everywhere

Posted by Michael Krigsman @ 6:42 am

Categories: End-user impact, News roundup, Project failures

Tags: Software, Information Technology, Flash Memory, Tools & Techniques, Management, Michael Krigsman

Software permeates even the most seemingly low-tech aspects of our lives. As we know, wherever software is deployed, bugs are sure to follow. Here’s a small sampling of recent software glitches from around the world.

Although none of these failures is earth-shattering, they all inconvenienced “victims.” I find the breadth and variety of these glitches absolutely amazing.

Seattle parking meters. Seatlepi.com reports that even parking meters can suffer software failures:

Read the rest of this entry »

August 8th, 2008

IT failures roundup: Emergency services around the world

Posted by Michael Krigsman @ 5:42 am

Categories: Availability and reliability, End-user impact, Government projects, News roundup, Project failures

Tags: emergency service, information technology, computer, productivity, michael krigsman

Today’s roundup focuses on public emergency services. Although these systems should never fail, here are several that did.

The impact of most IT failures is limited to inconvenience, delays, and higher costs. However, computer problems affecting police, fire, and medical response can directly cause loss of life and property.

Maryland 911. A boy almost drowned when the fire department didn’t respond following a faulty Verizon system upgrade. From the Washington Post:

Peter Lucht, a Verizon spokesman, said the disturbances in Prince William “stemmed from an unusual combination of factors” and were “not something that we usually see.” The company has worked closely with the county “to stabilize the system and ensure that [problems] won’t be repeated,” he said.

The county purchased its 911 system from Verizon in 2002, and it was installed in 2003, and fire officials said there were no problems until the May 28 upgrade. There have been no problems since July 12, they said.

Maine 911. Seven “system shutdowns” prevented callers from reaching emergency services. From the Kennebec Journal:

Between April and June, emergency communications systems at the Cumberland County regional communications center, the state dispatch center in Gray and the Penobscot County regional center had problems receiving 911 calls. The most severe problems were at Cumberland County’s system in Windham where a series of seven system shutdowns in April and May left callers getting no answer, in one case for as much as an hour.

In response, FairPoint installed a switch allowing dispatch centers to immediately transfer calls to a backup center when there is a problem, and a designated a separate telephone line to alert staff when there is a problem. The equipment was installed in the six dispatching centers in the state that had the type of equipment affected by the malfunctions.

London ambulance services. Failed computers caused London ambulance workers to fall back on pencil and paper call tracking. From the Evening Standard:

Ambulance staff were forced to record emergency calls with pen and paper and find addresses using A to Z after their computerised system crashed early yesterday morning and a computer failure at the Royal Free Hampstead NHS Trust means patients face delays and records have been lost.

The Royal Free introduced the system in June to reduce paperwork but since then it has crashed, leaving those waiting for operations and blood tests badly affected. When the same system was introduced at Bart’s and The London NHS Trust, cancer patients missed critical appointments because their records were lost.

A spokesman for the Royal Free said: “With change on this scale, it is inevitable that it will take time for staff to familiarise themselves fully with all the functions that they need to use but overall we are pleased with the progress being made.

“It is recognised that while staff are getting used to the system a small number of our patients may have to wait longer than expected in clinic.”

Australian ambulance services. System failure forced Queensland, Australia emergency services to record and track calls using whiteboards. From IT Failures blog:

Thorough investigations have been conducted as to the cause of each, with the following outcomes:

• Two of the outages have been attributed directly to human error. The first was an SQL update which had unexpected results, the second was a server which was inadvertently shut down.
• The third outage has been traced back to a fault with the replication software ‘Replistor’, which resulted in the primary database server rebooting.

[No humorous picture today, because there's absolutely nothing funny about these failures.]