IT Project Failures

May 12th, 2008

FBI: Counterfeit Cisco routers risk “IT subversion”

Posted by Michael Krigsman @ 6:05 pm

Categories: Vendor relationships, Government projects, IT issues, CIO issues, Security and privacy, Politics

Tags: Cisco Router, Router, Network, Information Technology, Cisco Systems Inc., Hardware, Routers & Switches, Federal Government, Networking, Network Technology

An internal Federal Bureau of Investigation presentation states that counterfeit Cisco routers imported from China may cause unexpected failures in American networks. The equipment could also leave secure systems open to attack through hidden backdoors. The scope of the problem is broad and results from a complicated supply chain originating in Shen Zhen.

From a narrow project failures perspective, network problems caused by this equipment should be treated as any other hardware malfunction. Of course, the entire concept of third parties using compromised hardware to infiltrate public and private systems in the United States is another matter entirely.

Faulty networking hardware can be a nightmare to troubleshoot and fix. For example, the U.S. Customs and Border Protection (CBP) location at Los Angeles Airport (LAX) suffered a failed router last year; the problem delayed 20,000 passengers before technicians successfully isolated and repaired the issue.

The following slides, pulled from the larger presentation, indicate how seriously the FBI is taking this threat to national security.

May 11th, 2008

Hosted software development: an emerging market [includes podcast]

Posted by Michael Krigsman @ 5:47 pm

Categories: Interview, Tools, IT issues, SaaS, PaaS, and SOA, CIO issues, Enterprise 2.0, Podcast

Tags: Software, Podcast, Environment, Emerging Market, Workflow, Tool, Development Environment, Tools & Techniques, Productivity, Software Development

Note: Click the player above to hear my podcast interview on hosted development with Jeffrey Walker and Jeff Leyser from Atlassian Software.

Hosted software as a service (SaaS) development solutions have become an important option for mainstream programmers. These products range from straightforward offsite storage, such as Amazon’s S3 service, to fully outsourced platform as a service (PaaS) application environments, like that which Salesforce.com provides in Force.com.

To learn about hosted software development, I asked two folks from Atlassian Software, which provides hosted group collaboration and software development tools, for an interview. Jeffrey Walker, the company’s President, and Jeff Leyser, Product Marketing Manager, briefed me on issues related to hosted software development.

What is hosted development?

Development systems consist of an integrated development environment (IDE), where programmers directly write and compile their code, along with other supporting tools. Two critical components sitting outside the IDE are the repository, where code is stored and its history maintained over time, and the defect tracking system, where developers track and manage bugs.

Hosted development solutions, such as Atlassian’s Jira Studio, provide these non-IDE components, along with other software development and collaboration tools, over the Internet.

In contrast, full PaaS systems, like Force.com, allow applications actually to run on their system. In other words, platforms include the IDE, supporting tools, have their own development language, and incorporate a run-time system.

What are advantages and disadvantages of hosted vs. local software development?

With hosted development, the hosting provider maintains the development infrastructure, relieving developers of this burden. This yields stable and predictable costs, as with other types of SaaS offerings. Ideally, hosted environments should also embody development best practices, such as incorporating tools to make code reviews easier and faster.

Disadvantages of hosted development include perceived concerns around the safety of intellectual property. Although this issue exists in any remotely hosted SaaS environment, software companies may be reluctant to entrust their key intellectual assets to a remote service provider. Another issue is customization, because SaaS development tools are generally less configurable than locally managed, on-site systems. Similarly, it is easier to integrate internal systems, such as LDAP directories, with local tools than with hosted systems.

When is hosted development most appropriate?

Geographically distributed teams are an obvious candidate, particularly where organizations combine offshore development with an onshore team. Hosted development gives both local and remote groups straightforward access to code and a standardized set of tools anywhere the Internet is available. Hosted solutions also simplify implementing common workflow processes across a distributed team.

How is the workflow and process different with hosted development?

The initial setup and infrastructure planning are quite different with hosted development. However, the daily programming workflow is the same in both local and hosted development environments. Features and functions of the tools drive workflow, regardless whether the tools are behind the firewall or installed locally.

Any comments on Salesforce.com’s hosted development platform?

Answered via email: It will be a cold day in Hell before someone builds a serious piece of software totally unaffiliated with SFDC on Force.com. I am sure SFDC can name examples but I would just point to the failure of the AppExchange after huge promotion and investment….Be careful in comparing a hosted development environment that allows for a myriad of types of projects, like ours aspires to be, with ones that promote a self-serving agenda or proprietary infrastructure.

May 7th, 2008

Government IT failures: “Room for improvement” [interview and podcast]

Posted by Michael Krigsman @ 2:02 pm

Categories: Government projects, Project management, Interview, IT issues, Project strategy, CIO issues, Podcast, Project portfolio management, Politics

Tags: Project, Podcast, Agency, Information Technology, OMB, Project Management, Tools & Techniques, Strategy, Advertising & Promotion, Government

Given the size, scope and frequency of government IT failures, it’s important to understand the dynamics underlying these projects. To learn more about federal IT, I interviewed two federal systems experts from CA, developer of the Clarity project portfolio management (PPM) solution.

Gil Digioia is Vice President of Federal Project Portfolio Management Sales for CA Clarity, and possesses over 15 years working delivering software solutions tailored to the specific needs of federal agencies and solution integrators. Jose Mora is Sr. Director for Product Marketing for CA Clarity Project and Portfolio Management solution, where he is responsible for all IT governance and public sector-related marketing activities including product strategy and positioning.

The OMB publishes a quarterly list of high-risk IT projects as part of its investment oversight function. Why is this list important?

It’s an extremely important means for the Office of Management and Budget (OMB) to provide visibility and transparency regarding IT projects to the agencies. In addition, citizens want to know how agencies allocate and invest funds, and whether they are achieving their deliverables. Having a list of projects in the red zone is also critical to making decisions.

Do government agencies pay attention to the list?

Absolutely. The most important thing about this list is bringing attention, and drawing focus, to where agencies will direct their spend and plans over the year. A lot of attention is placed on bringing these high-risk projects into the green.

In addition, when an agency has a change, such as new CIO or director wanting to make an impact, the list is often used to bring about quick success.

Why do so many government projects fail?

There is room for execution improvement regarding the triple constraints of scope, time, and budget. The big reason is lack of “critical corrective action” from high-level decision makers within the organization. This can result from either lack of decision-making or leaders who don’t have the proper information to make decisions that ultimately impact the project.

Requirements also tend to change after projects have been awarded, and are often different at project conclusion from what was specified in the original proposal. These changes tend to disrupt project workflow and collaboration. Such challenges pose particular difficulties for organizations that don’t have a repeatable governance process in place or lack the proper technology to react easily to those changes.

Problems can arise at the project management level, the executive decision-making level, and with technology. For example, problems are sometimes caused by legacy systems that can’t adapt to the rapid changes in information these organizations face.

Are there differences between public sector and private sector projects?

The biggest difference is profitability on the finance side. In the private sector, cost reduction is important but they’re also looking to drive profitability.

From the perspective of project management and program management behavior, however, the federal government is definitely ahead of the private sector. Various mandates on government projects, such as OMB and enterprise architecture requirements, are much further ahead than what is typical in the private sector. For example, “earned value” is still a novelty in the private sector but it’s important in the government.

How can federal organizations evaluate IT projects to prevent failure?

Earned value management, which the OMB mandates, is a solid and rigorous project management technique. It captures the fundamentals for evaluating a portfolio, including analysis of baseline costs, performance data, schedules, resources and skill sets, outcome measurement, and examining reporting requirements. All these provide a solid foundation for evaluating one project versus another.

Aside from systematic upfront evaluation, what steps can federal managers take to ensure successful projects?

Selecting the right governance structure is important, which means examining how an agency makes decisions on its program portfolio and ensuring this process is repeatable and established throughout the organization. This process component is fundamental.

It’s also important to define comprehensive criteria for evaluating projects and to ensure that criteria includes all critical areas or details that could impact earned value reporting back to the OMB. The earned value requirement established by the OMB is a step in the right direction.

Accurately capturing and reporting project data is another key area. Although this sounds simple, we often see complicated, homegrown legacy applications for project management, resource management, and tracking time or cost. All these disparate systems create confusion and complexity, making it difficult to track accurate data.

Project improvements also mean mobilizing and empowering the proper infrastructure of people needed to make and communicate decisions throughout the organization. Leadership is required to make difficult decisions such as killing a project, delaying a project, or simply not funding a project.

To be successful, start with sound processes, as opposed to throwing a tool or solution at the problem. Get your processes in place, understand your risks, and then create a platform where you can organize the people who need to execute, in a manner that’s repeatable, process-driven, and has visibility from executive management. Take on smaller wins that deliver value quickly instead of attempting a big bang.

[Thanks to Joan Levy from Blanc & Otus Public Relations for arranging this interview.]

May 6th, 2008

IT politics killed White House email project

Posted by Michael Krigsman @ 7:38 am

Categories: Project failures, Government projects, Project management, IT issues, CIO issues, Politics

Tags: Information Technology, White House, Act, E-mail, Online Communications, Michael Krigsman

Data archiving in the White House is a serious business mandated by the Presidential Records Act of 1978, which was passed following the Watergate scandal.

The Act requires the White House to maintain an historical archive of its activities, policies, and decisions. Despite this law, the White House email archiving system is a model of poor IT practice and has been called “primitive,” “inadequate,” and “not robust.” The system fails to fulfill its most basic requirements: enabling reliable backup, storage, and restore capabilities.

Email backup process. Quoted in a report by the House of Representatives Committee on Oversight and Government Reform, White House CIO, Theresa Payton, described how White House emails are archived using a manual method called “journaling:”

Under this process, a White House staffer or contractor would collect from a “journal” e-mail folder in the Microsoft Exchange system copies of e-mails sent and received by White House employees. After retrieving copies of these e-mails, the White House staffer or contractor would then manually name and save them as “.pst” files on various White House servers.

Former White House CIO, Carlos Solari, characterized the process:

[A]s a ‘message collection system’ even though we all understand that it hardly qualifies as a ’system’ by the usual IT definition.

In a memo to Acting CIO, John Straub, in 2005, IT manager, John McDevitt, described the ad hoc system:

The current email archive process depends on manual operations and monitoring, standard operating procedures do not exist, automated tools that support the email archive process are not robust, and there is no dedicated archive storage location.

The report points out at least three fatal flaws with the manual email archive process:

• Risk of data loss
• Risk of tampering
• Inability to verify system functionality

Email archiving project failure. In 2003, the White House initiated an Electronic Communications Records Management System (ECRMS) project to automate email archiving. Booz Allen Hamilton was contracted to design the system and Unisys was engaged to test and implement it. According to the internal government program director for the project, John McDevitt, the project was completed in the spring of 2004:

According to Mr. McDevitt, this design was presented to the White House Counsel, the White House Office of Records Management, and counsel in the Office of Administration “for their concurrence” in the spring of 2004. With Unisys serving as the contractor for the implementation phase, the White House undertook “[s]ystem configuration, testing and tuning” through 2005. In early 2006, standard operating procedures were developed. In March 2006, the White House Counsel, the White House Office of Records Management, and OA counsel were briefed on the system, and in July of 2006, they were briefed “on the search and retrieval capabilities of the ECRMS solution.” Mr. McDevitt stated that the project was “ready to go live” on August 21, 2006.

Although the ECMRS was ready for use, current White House CIO, Theresa Payton, terminated the project in 2006 because:

“[t]he system would require 18 months to ingest the existing backlog of messages in the Microsoft Xchange system” and “[t]he system offered users no option to distinguish between Presidential records and political or personal materials.”

The National Archives responded with objections to these reasons, suggesting they did not present sufficient cause to abandon the completed project and revert to manual, and therefore unreliable, email backup techniques.

THE PROJECT FAILURES ANALYSIS

Politics and other personal (and organizational) agendas are usually to blame for IT failure. By any reasonable measure, the guardians of White House email used poor IT practice as a tool to circumvent applicable law, avoid disclosure, and maintain control over sensitive data.

Since IT underlies most modern business and government processes, decisions about software, infrastructure, and deployment can have broad ramifications for how leaders execute business strategy. In this case, it appears IT leadership deliberately made poor technical decisions to achieve a specific political strategy.

Update 5/6/08 5:00pm EST: To gain further insight into this situation, I spoke with David Gewirtz, author of the book Where Have All the Emails Gone?. Here’s what David said:

White House email is broken. Their email archiving system is wildly inadequate to the point of negligence. Management of computer assets like laptops, flash drives, and BlackBerrys is completely non-existent.

Email in the White House needs to be fixed. Not because we want to give Congress a bigger stick with which to beat on Presidents, but because some really bad things could happen if it’s not fixed. There are technical issues and concerns, plus security issues and concerns that blast through the political rhetoric and even party affiliation. The practice of archiving is a technical act, while the practice of disclosing is a political or policy act.

We need to make sure we archive the White House email traffic, but that doesn’t mean confidential information must be disclosed to opposing parties or the general public.

Finally, there is breaking news out of the White House today. The White House has responded to Judge Facciola’s request for disclosure of email messages during the first term of the Bush administration. I’ve just gotten those court documents and am working my way through them now. The gist of them seems to be that the White House does not believe further document recovery is warranted. I’m going to be working my way through the full document set and will publish an analysis of it, probably tomorrow.

May 2nd, 2008

Sleazy sales and the Rockwell Retro Encabulator

Posted by Michael Krigsman @ 1:32 pm

Categories: Vendor relationships, IT issues, CIO issues

Tags: Rockwell, Sales Strategy, Sales Force Management, Corporate Communications, Sales, Marketing, Michael Krigsman

Many project failures can be traced to a vendor making outlandish claims which the customer accepts hook, line, and sinker. In these situations hope, greed, fear, inexperience, and denial come together as ingredients for a spectacular flame-out.

With that mind, take a look at this sales pitch for the Rockwell Retro Encabulator. Now tell the truth, did you believe the pitchman in the video? It’s amazing how even the most unvarnished nonsense can be delivered with such conviction.

May 2nd, 2008

10 reasons for IT failure

Posted by Michael Krigsman @ 8:32 am

Categories: Project management, IT issues, Project strategy, CIO issues

Tags: Project, Information Technology, IT Leadership, Complex Project, Strategy, Leadership, Management, Michael Krigsman

Readers of this blog know IT initiatives generally fail for business, organizational, or cultural reasons. Sure, technology screw-ups occur all the time, but that’s one of the realities to be managed. Success or failure ultimately depends on how project leadership manages the full range of technical- and non-technical issues.

Blogger and enterprise architect, Mike Kavis, who’s obviously been through some battles, has created ten guidelines describing critical areas of weakness in many projects:

1. Poor Communication. Enterprise projects usually impact a large amount of people. This requires constant communications to all levels of people throughout the organization. A strong communication strategy can help with this.
2. Underestimating or ignoring impact of change. This is another way of saying poor change management. People need to know WIIFM (what’s in it for me). Resistance to change can kill any project. Your initiative must have a champion who carries a lot of clout.
3. Lack of Leadership. IT Leadership requires excellence in three key areas: Technology, Business, and People. If the leadership is missing any of the three components you are doomed.
4. Lack of strong executive sponsorship. For these projects to succeed you must have somebody high up in the organization with a lot of clout.
5. Poor project management. Often, large enterprise initiatives have a ton of logistics that need to be identified and managed accordingly.
6. Poor Planning. This could also fall into a category of unrealistic expectations. Initiatives like SOA require a well thought out strategy. Many IT shops do not have the patience for this and rush into their project head first without a clue of how to actually accomplish their goals.
7. Trying to do it cheap. Organizations want it all, but they don’t want to invest the time and money. I have seen many projects get completed using this strategy, but they almost always run over budget, are late, are missing many features, and have many various quality or process issues due to the quick-n-dirty approach.
8. Lack of technical knowledge. You wouldn’t ask me to remove your appendix so why would you have somebody with little or no knowledge of the technology at hand lead your enterprise initiative.
9. Lack of sound business case. You can get all of the other issues right but if your solution has no business context then you are wasting your time.
10. Poor vendor management. Somebody hires a high priced group of consultants and let’s them run wild. You should make sure that what they build meets your requirements, your standards, your needs, and your timelines.

Whenever I blog this kind of list, some commenters invariably say the whole thing is obvious common sense and therefore not worth consideration. If so, then why do so many projects fail precisely due to items on this list? Complex projects are hard to get right, which is why IT failure remains a serious issue.

Successful leaders create project success on the foundation of skillfully managing people, process, and technology. While this perspective may appear obvious, the experience and wisdom needed to make IT projects successful is not common at all.

May 1st, 2008

Google Calendar is down

Posted by Michael Krigsman @ 12:58 pm

Categories: Enterprise 2.0, Availability and reliability, Failure 2.0

Tags: Google Inc., Google Calendar, Michael Krigsman

Update 5/1/08 4:4o pm EST: Seems to be back up now.

Google Calendar is down. Glad my meetings aren’t stored there. Cool error page, though. Love all the languages.

April 30th, 2008

Volvo recall due to airbag software problems

Posted by Michael Krigsman @ 1:04 pm

Categories: Project failures, IT issues

Tags: Software, Volvo, Inflatable Curtain, Tools & Techniques, Management, Michael Krigsman

Swedish automaker, Volvo, recalled 65,000 cars due a software problem related to side impact airbags. I spoke with Volvo spokesperson, Maria Bohlin, who told me the problem was discovered by Euro NCAP, a European automotive testing agency, which found the “inflatable curtain deployed milliseconds too late.” She added that Volvo would examine whether their software and testing routines are sufficiently robust.

Firehouse.com offers excellent technical information about Volvo inflatable curtain (IC):

The IC is a long, flat airbag which deploys down from the roof rail and extends from the A-pillar to the C-pillar, covering the front and rear side window areas from the roof to the top edge of the door. One side-impact crash sensor is mounted low on the inside of the B-pillar. A second crash sensor is mounted low on the C-pillar at the seat-cushion level. The IC inflates only on the side where impact takes place. A pressurized gas cylinder contains 95% argon and 5% helium gas that fills the Inflatable Curtain airbag.

This photo shows the actual stored gas cylinder used in the Volvo system:

Airbags are complex devices that must deploy within a fraction of a second, between the time a vehicle strikes an external object and the rider hits the front or side of the car interior. This diagram illustrates the kind of logic used in airbag software:

This is not the first time an auto maker recalled cars based on software issues. Earlier this year, Ford recalled 470,000 Mustangs because the “passenger-side frontal airbag could potentially deploy with greater-than-allowable force for a petite, unrestrained passenger.”

Software permeates our lives in the most unexpected places and sometimes it goes awry. Volvo and Ford have learned that lesson the hard way.

April 30th, 2008

CNBC: SAP America’s president should offer media training

Posted by Michael Krigsman @ 6:11 am

Categories: SaaS, PaaS, and SOA, SAP

Tags: CNBC, Media Training, SAP AG, Advertising & Promotion, Marketing, Michael Krigsman

During an interview on CNBC, SAP’s CEO Bill McDermott, said customers buy his software to accomplish three goals:

• Making money
• Cost reductions
• Maintaining compliance

In a moment of what some might consider to be irrational exuberance, McDermott described SAP as the only software company able to offer these things to customers. I suspect some SAP competitors might take issue with his comment.

One interviewer jokingly asked McDermott whether he offered media training, since he turned all the negatives into positive comments. Fellow Enterprise Irregular, Charlie Wood, was also impressed. His post includes the full clip.

On a more serious note, SAP announced that Business byDesign, their software as a service (SaaS) offering, would be delayed 12-18 months. Larry Dignan and Dennis Howlett covered that story.

Business by Design represents a strategic shift for SAP in the years to come, so I think the delay is only a temporary setback. As Vishal Sikka, SAP’s chief technology officer, told me:

“[Business byDesign represents] a profound shift we are trying initially in the mid-market; we are being cautious, we are being conservative, we can afford to, and we are going to get this right.”

See also: Exclusive interview / podcast with Vishal Sikka, SAP’s chief technology officer

SAP has been touting the product to its ecosystem, and partners building their business around this new initiative will be disappointed with the delay.

April 28th, 2008

SAP’s CTO: Business knows ‘what’, IT knows ‘how’

Posted by Michael Krigsman @ 3:28 am

Categories: Implementation, Interview, IT issues, SaaS, PaaS, and SOA, CIO issues, SAP, Naked IT, Podcast, IT extinction

Tags: CTO, Line Of Business, Information Technology, Packaged Application, SAP AG, Business byDesign, Strategy, Management, Michael Krigsman

The Naked IT interview series talks with innovators about the evolving relationship between IT and business. Please listen to the audio podcast and enjoy the brief excerpts below.

Vishal Sikka is SAP’s Chief Technology Officer. Reporting directly to CEO Henning Kagermann, Vishal is responsible for driving technology and architecture strategy across SAP’s product portfolio. As CTO, Vishal also leads the company’s forward-thinking efforts around emerging technologies and is responsible for mapping SAP’s next-generation architecture. He holds a Ph.D. degree in computer science from Stanford University and his experience includes research in automatic programming, information and application integration, and artificial intelligence at Stanford, Xerox Palo Alto Labs and two startups.

Vishal and I spoke about a broad range of enterprise software issues, including his role as CTO of SAP, service-oriented architecture (SOA), relationships between business and technical organizations in the enterprise, IT project failures, SAP’s acquisition of Business Objects, and Business byDesign.

If you’re interested in CIO or CTO issues, give the podcast a careful listen; it’s well worth your time and attention!

On being CTO:

My primary job is to define and articulate the roadmap for our products and technology, bringing coherence and uniformity to the way we govern our architecture and over-arching product design.

On packaged applications:

The basic idea of a packaged application is one size fits all, serving a wide variety of customers. You get economies of scale from packaging that application, functionality, and so forth….Having a packaged application suite that simultaneously fills the need across that broad diversity of industry, country, nature of business…is a very, very difficult problem. There have been generations of companies that have tried to do this and failed. Simultaneously reaching for breadth, depth, and ease of use killed these other companies.

On IT / business alignment:

Lines of business know the what and IT knows the how. Companies in which this “what / how” distinction is broken tend to have difficulties. You have to empower the lines of business and the IT guys….trusting the IT guys know the how and lines of business can articulate the what….The business [should] tell IT the nature of the problem to solve; IT [should] then bring in tools and platforms to [address them].

You [develop] strategic IT by bringing the line of business and the IT guys together to the table [with] mutual respect and empowerment.

On IT extinction:

That is just nonsense.

On reducing IT failures with enterprise SOA:

[SAP combines] IT simplification (technical visibility) with business transformation. Implementation maps [along with] the enterprise services repository and its process capabilities…help get solutions get adopted very quickly [and with lower risk]. The services repository gives [stakeholders] a very public and visible way of seeing how their business relates to the software.

[Stakeholders] can also [share] with other business process experts to get a sense of what others are doing, which Hasso [Plattner], our founder, refers to as “doing business with an open window.” Seeing what other people have done, sharing experiences, [provides] extra visibility into [how the software relates] to the business, making the entire process much more seamless, and reducing project failure.

On the Business Objects acquisition:

We closed the strategy to execution loop, which CEOs , CFOs, CIOs, and heads of operations are badly missing….Closing that loop between strategy and execution is something that Business Objects really enables us to do.

The user-centric and the information-centric aspects are fundamental….We want to be in activities that business users [perform]; that are not directly operational, but informational, in nature. We are going to get there faster than any of the other guys.

On innovation in enterprise software:

If you have applications that serve core business processes and live for ten or fifteen years, how do you bring innovation that is non-disruptive to customers? We deliver innovation in consumable packages twice a year in enhancement packs, which also include technology improvements; this is a non-disruptive roadmap for [customers] to continuously innovate. [It’s] a mechanism for delivering innovation while preserving stability; of delivering insight while preserving execution.

On Business byDesign:

Salesforce.com talks about adapting and so forth, but doesn’t cover even a tiny fraction of the breadth of [SAP’s product suite]. On-demand to us is an interesting delivery mechanism…. [However, Business byDesign represents] a more profound shift….

A few years ago we launched an effort to rethink how we build the core software;…it was our generational look at what has to happen inside the core processes. Business byDesign is our mid-market delivery of that vision…It represents a fundamentally modular way of building software that no one in the industry has ever dreamed of building. It is designed for the mid-market but has implications beyond that.