November 6th, 2009

Event processing means more than 'speeding up' existing systems

Posted by Joe McKendrick @ 8:43 am

Categories: Business ROI, Data managemetnt, Enterprise Architecture, Event processing, Management, SOA Events, SOA Surveys and Research, business process management

Tags: Event, Service-Oriented Architecture (SOA), Web Services, Pricing, Strategy, Application Integration, Middleware, Enterprise Software, Software, Marketing

Complex event processing — now made possible by service-oriented architecture principles — represents the next stage of business intelligence. However, much work needs to be done to reach this capability.

Complex event processing requires a different mindset and skills

A ebizQ’s latest SOA in Action conference, I had the opportunity to moderate a session with Gartner’s Roy Schulte, CalTech’s Dr. Mani Chandy (CalTech), and IBM’s Frank Chisolm in an informative discussion about applying event processing as a strategy for businesses seeking to remain competitive in the years ahead.

However, Roy cautioned, event processing capabilities don’t just automatically pop up, even among companies with the most advanced BI infratstructures. “The way you get your systems to be more smart fast and agile is by having the systems designed correctly, and in most cases that means more use of the event processing design methodology,” he says.  “You can’t just take a conventionally designed system and just speed it up to accomplish the goals that people want to do.”

While the technology now exists to build CEP, the methodology requires a different mindset among companies. “The limitation that we have today is that there are not enough people around who understand how to design systems that operate in this fashion,” Roy says. “They don’t understand continuoius intelligence or complex event processing.”

Complex event processing requires continuous streams of information from multiple sources. The good news is that CEP need not be so complex, and, in fact, over the next few years, systems that sense and respond to events will be as commonplace as business intelligence systems are today.

Mani, considered one of the early visionaries of complex event processing, said the “PC-cubed” formula (three Ps and three Cs) will drive CEP forward over the next few years:

• Price – The price of managing data sources will continue to drop.
• Pervasiveness – Sensors, such as mobile phones, have become pervasive.
• Performance – “Enterprises have access to immense computing power that can be harnessed through event processing,” Mani says. And now, “parallel, distributed, and cloud computing create ideal environments for event processing.”

• Celerity - “Businesses and consumers demand swift action,” Mani points out. “You expect to be notified immediately if your plane is late.”
• Connectedness – The world is more interconnected. Your company may need to respond immediately to an earthquake in China, a flood in India. Event processing applications help detect events all over the globe.”
• Complexity – “Businesses have become more complex, and expect IT to help with increasingly complex problems.”

As if laying out the case for complex event processing as “PC” doesn’t clarify enough, Mani also explained how a mnemonic — A, E, I, O, U (but not sometimes Y) — describes the CEP phenomenon:

• A — Adaptability: “The event pattern has two advantages, one is loose coupling for application integration, and the other is sense and response,” Mani said. “App integration because producers and consumers are coupled in a loose way without knowing about each other. Its easy to add or change the producers and consumers of a system. With the sense and respond aspect, an example is scheduling railroad crews — a complex problem, a sense-and-response problem.  Because unscheduled events happen all the time, smart railroads are using event processing to adapt.”
• E  — Exceptions: “Computers have to analyze torrents of data to extract nuggets,” said Mani. “These nuggets are the events that require a response. A characteristic of smart people and smart systems is that they mange by exception.. they perform continuing operations effectively, bit they continue to detect and respond exceptional situations. Event processing helps separate the critical from non-critical.”
• I  — Instrimentation: “Successful businesses manage exceptional events successfully,” according to Mani. “Event processing is used to instrument and monitor the exception and the normal. You will see a rapid rise in business instrumentation and event processing for to improvement of business activity in the next decade.”
• O — Outside: “1960s-90s enterprise IT dealt with mainly IT inside the enterprise. Now the enterprise is responding the events externally,” said Mani. “The enterprise monitors actions by the government, its competitors, its suppliers, and its best customers.  The ability to sense and respond to events out side the enterprise using event processing is a significant competitive advantage.”
• U — Unanticipated events: “Enterprises develop event process applications to handle certain types of that they expect, and must also deal with conditions that they don’t expect,” Mani explained. “Any significant deviations are detected by an event processing application which then sends information about this deviation to appropriate people before the analysis.”

Dr. Mani Chandy and Roy Schulte have just puiblished a new book on the subject, entitled “Event Processing - Designing IT Systems for Agile Companies.”

October 14th, 2009

How event driven architecture changes the SOA service flow

Posted by Joe McKendrick @ 7:14 am

Categories: Event processing, Links, SOA Surveys and Research, Web Services, business process management

Tags: SOA, Service-Oriented Architecture (SOA), Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

Does event driven architecture (EDA) represent the next phase of SOA?  This is a subject of continuing debate, but Udi Dahan makes the case for EDA as the next logical stage of SOA. In a recent post, he connected the dots between SOA and EDA, suggesting that EDA will shift the inherent nature of SOA.

Dahan makes his case thusly: SOA is currently based on a “commonly used request/response communication pattern of service consumer to service provider in SOA,” in which the consumer delivers the command, and the provider service responds accordingly. “Commands are often named in imperative, present-tense form—for example, ‘update customer’ and ‘cancel order.’”

With EDA, this relationship gets reversed, he points out:

“In EDA… Consumers do not initiate communication in EDA; instead, they receive events that are produced by emitters. The communication is also inherently unidirectional; emitters do not depend on any response from consumers to continue performing their work.”

Developing an architectural approach that employs both SOA and EDA principles will go a long way toward better so-called “business-IT” alignment, Dahan observes. The fusing of the two approaches may be the key. Either approach alone won’t do it. As Dahan illustrates:

“Architects can explain to the business the ramifications of their architectural decisions in ways that the business can understand—’There might be a couple of seconds during which these two bits of data are not in sync. Is that a problem?’—and the answer to those kinds of question is used to iterate the architecture, so as to bring it into better alignment with the business.”

SOA and EDA have been moving closer in recent years, as companies start to understand the value of event processing to ongoing operations and opportunities.

September 29th, 2009

SOA, event driven architecture will be in smart systems everywhere

Posted by Joe McKendrick @ 7:33 pm

Categories: Event processing, Links, SOA Surveys and Research, business process management

Tags: SOA, EDA, Service-Oriented Architecture (SOA), EAI, Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

In recent years, we’ve been hearing a lot about the rise of event-driven architecture (EDA), and how this will factor into SOA efforts. This combination may form the foundation of emerging “smart systems.”

CalTech’s Dr. K. Mani Chandy, one of the pioneers of EDA, and author of Event Processing: Designing IT Systems for Agile Companies, says we will be surrounded by SOA and EDA in the years to come. In a new interview with Peter Schooff, Chandy explains:

“I see a great opportunity for both [SOA and EDA] in the next, I say, 20 years starting now. …I really see them being used in all aspects of daily life. I mean management of food, water, energy, health, security and logistics… And what we call smart systems. A lot of talk about smart systems and smart system architectures are fundamentally based on principles from EDA and SOA. And the benefits of these event-driven architecture applications will be directly visible to the business, they’ll be directly visible to the customer and so I think acceptance of these applications and demand for these applications will grow virally and so I see a great future for them.”

Chandy has a formula that illustrates how and why demand for EDA will grow in enterprises in the years to come: “PC-cubed,” for push for ‘price, pervasiveness, performance” accompanied by demand from “connectedness, celerity, and complexity.” The need to process complex events will not only arise in large enterprises, but also in the consumer space as well. “We see really complex events where you’re trying to detect patterns of stock prices and relate them to commodity prices. But you also see complex events in consumer applications where you’d like to know when a given item becomes cheap with one vendor versus another,” Chandy points out.

EDA has been around in various forms since the 1970s, Chandy explained. It’s roots can be traced back to enterprise application integration and sense and response systems. “EDA appeared in the 1970s in message queuing systems and later in enterprise service buses, and this is the EAI ‘parent,’” he said. “The other ‘parent’ is sense and response. In the last decade, many companies have developed sense and respond applications in finance particularly in trading. Now however, sense and response systems are being used in every aspect of life including management of water, food, energy, security, health and so on.”

August 3rd, 2009

A brief history of complex event processing

Posted by Joe McKendrick @ 6:09 pm

Categories: Event processing, General, Links, Vendor Watch

Tags: Club, Paul Vincent, Joe McKendrick

Confused about the CEP market?  Join the club. Paul Vincent has prepared an excellent timeline of the growth of the complex event processing (CEP) market, documenting the entree of major vendors. And, as Paul says, it will be interesting to see how the chart continue to evolve over the coming year. Notice how everyone is poised to spring into 2010.

Source: Paul Vincent, TIBCO

June 11th, 2009

Why does 'complex event processing' have to be so complex?

Posted by Joe McKendrick @ 4:00 pm

Categories: Business ROI, Event processing, SOA Events, Vendor Watch

Tags: Event, Financial Services, Investment, Financial Planning, Finance, Joe McKendrick

I’ve always felt that “complex event processing” is a term that scares people away. Perhaps there needs to be a softer way to describe what this thing is — perhaps just plain old “event processing” will do? Or perhaps “sense and respond, times 100,000″?

You don’t need to be a state-of-the-art, completely online organization to move to event processing. David Olson, director of CEP product marketing for Progress Software’s Apama unit, says from a technical standpoint, the fundamental system requirement for event processing is having a server to capture and correlate events, apply appropriate rules to an event stream, and send the information to a dashboard or portal.

But the success of an event processing implementation is a management challenge, he says.  “One of the more complicated endeavors that people have to figure out is what events make sense to them, and what types of patterns do they need to look for?”

I recently had the opportunity to join David, along with Brenda Michelson, principal with Elemental Links, in a rousing roundtable discussion on the impacts of event processing in what has been a very eventful year for businesses. (Full transcript available here.)

For example, financial services has been at the center of the economic storm over the past 18 months. If these companies had more advanced capabilities to process and analyze key events, they might have been able to head off many of the slips and slides that led to the financial crisis of 2008, David and Brenda agreed. Still, there are many lessons being gleaned from the whole experience that will benefit all industries for years to come.

“I think [Complex Event Processing] could have been used in some situations that got us into this financial crisis,” says Olson.”Perhaps we should have employed CEP a few more years ago in order to help stem the tide of what has happened. But there’s certainly a lot of learning that’s been going on as people go back and research what has happened, to figure out what kind of rules they should put in place in the future to make it happen again.”

Brenda pointed out that millions of events occur across enterprises every day — from stock prices bouncing to soda cans shifting on a pallet in a warehouse. The challenge is being able to capture and process the events that have the greatest impact on the business. “Event processing is what discerns is if that thing that happened is notable,” she explained:

“Is it important to me? Is it important to my business? Do I need to act on it?  And is that notability by itself, or perhaps its notable because of a couple of other events that are going on? Is it just my business that’s bad, or is it the entire credit market that’s tanked? And then once you discern that notability.. you decide what am I going to do with that event next?  Am I going to forward it along into an event channel? am I going to trigger some kind of downstream action?”

David elaborated on that theme and pointed out that “its not sufficient just to be able to capture the events and perhaps store them for future use, since the volume and the velocity can be quite great in certain circumstances.”

There is great potential being demonstrated in financial services in terms of unearthing fraud, David points out. “One of the exciting areas in CEP, especially in capital markets is the whole notion of surveillance, where there are a fair amount of rules that look for abnormal patterns of activity,” he says.

“For example, one of our customers uses it to monitor actions that could indicate insider trading, or joint trading opportunities where two traders trade large volumes of a symbol, but not large enough that it tips anybody’s radar to abnormal activity.But when these joint trading activities occur either ahead or behind a news feed, that could indicate to us there’s unusual activity going on here, and we should do something with this particular symbol that’s being traded. In the past, trying to catch those types of activities could take months of auditing in order to figure out that something bad happened. And in the capital markets space, minutes of something bad happening could be billions of dollars.”

March 19th, 2009

SOA without service-enabled applications?

Posted by Joe McKendrick @ 8:00 am

Categories: Business ROI, Enterprise Architecture, Event processing, General, Management, SOA Surveys and Research, Vendor Watch

Tags: Oracle Corp., SOA, Service-Oriented Architecture (SOA), Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

“SOA Possible Even Without Service-Enabled Apps.”

This is a statement that goes against the conventional wisdom, so, being a fan of things that go against conventional wisdom, I checked out this Q&A interview with Shailender Kumar, vice president of Oracle Fusion Middleware for Oracle India, to see what his thinking was. I wasn’t dissapointed.

As Kumar put it, the idea that SOA requires that participating applications be service-oriented is a “myth.” Most IT shops, in fact, will have a mix of approaches. There will be legacy systems, and there will be “modern” systems, there will be all kinds of middleware and messaging brokers. As he explains it:

“If you have an application that is service-enabled, and a whole bunch of applications that are not service-enabled, you can still connect these by deploying adapters. Once [people] realize that, they start to see where SOA can fit in bringing connectivity between diverse transaction engines.”

Oracle’s strategy is to position Fusion as the platform that will bring together a lot of diverse assets from across the enterprise into a service layer, and, not surprisingly, this is reflected in Kumar’s statement. But unless an organization throws out all its systems and starts entirely from scratch these days, most SOA efforts will be very ungainly and unique contraptions — and that’s okay. In surveys I have seen and conducted, even the most advanced SOA-savvy companies have less than 20% of their portfolios SOA-ready. And, of course, JBOWS is the predominant architecture at this point. And that’s okay, too. It’s a stage in evolution. And in all likelihood, there will be no compelling need to service-enable 100% of everything.

But SOA is in a lot of places, Kumar also reminds us. For example, every time we order from Amazon (an Oracle Fusion customer), the order is processed via a service-oriented framework.

December 29th, 2008

Ten examples of SOA at work, circa 2008

Posted by Joe McKendrick @ 3:04 pm

Categories: Business ROI, Case Studies, Enterprise Architecture, Event processing, General, Links, Management, Web 2.0-Enterprise 2.0, Web Services

Tags: SOA, AbeBooks.com, Con-Way, Service-Oriented Architecture (SOA), Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

What has changed about SOA over the past year? Well, many companies have finally moved on beyond the experimentation and pilot stages. As Judith Hurwitz would put it, there are a lot more SOA implementations now than ever before. In writing the next release of her team’s book, SOA for Dummies, Judith observed they were able to document 24 live examples across nine verticals. Just two years ago, for the first edition, most people were still trying to grasp what SOA meant.

In that spirit, here are some stellar examples of where and how SOA made a difference for some companies in 2008. These stories are culled from my “SOA in Action” site over at ebizQ, where I document working examples of service oriented architecture.

To modernize ’special’ applications with limited reuse potential. Allstate Insurance moved to SOA to standardize a generation of “special” systems it had built over the decades. Anthony Abbatista, Allstate’s vice president of technical solutions said the company managed to automate most of its systems and processes in the 1960s and 1970s with hand-crafted and hand-coded “special applications.” Everything from policy administration to underwriting was captured on the company’s array of mainframe computers. However, the company needed to embark on a more consolidated, service oriented architecture if its systems were to continue to deliver value. Areas being modernized include claims processing, which was spread across nine different mainframe-based silos tied together by custom-built middleware. The company employed an enterprise service bus to handle claims processing through a single interface across the entire enterprise. Another initiative was to establish a common integration layer for the company’s fast-growing data environment.

To address ’simple’ problems, and grow from there. OppenheimerFunds first leveraged SOA to get duplicate data entry under control. Then things took off from there. The company took this initiative to think long term, and not only deal with the data entry, but then to also launch a long-term effort to take down the silos and eliminate redundant processes. After an address-change service was leveraged, another service was created to update bank information, then to enable electronic imaging of paper-based documents. Today, there are 22 legacy applications used, in various combinations, by customer service agents. (Source: ITWorld)

To underpin new cloud-based offerings.  Service-now.com, begin in 2004 as an on-demand IT Service Management solution provider, recently launched a cloud-based offering that combines the best of everything — ITIL v3 with Web 2.0 technology and SOA — to provide a rich user experience to address a firm’s problem management needs. For Service-now.com, SOA means extreme competitive advantage.  According to the company’s founder and president, “there were no alternatives, no decisions to be made. There was no other way than with a SOA mindset.” (Source: CIO)

To support Extreme Transaction Processing. The ability to capture huge data flows is what’s been powering business at a leading Canadian online bookstore site, AbeBooks.com, which has harnessed the power of XTP as part of its service oriented architecture. Abebooks.com manages a database of more than 110 million new, used, rare, and out-of-print books via Web services links to 13,500 booksellers. Previously, transactions hit the back-end databases directly. “We’re in the first stages of design and implementation of service-oriented architecture,” according to Leith Painter, manager of development at AbeBooks. “We’re sponsoring it from an IT level. We’ve got some initial services we’ve developed in a design phase, and we’re currently developing design principles.” (Source: SearchSOA)

To modernize a 40-year-old system. Until recently, both Lufthansa and United Airlines — part of a consortium called the Star Alliance — were using a 40-year old reservation system, written in Assembly language. The airlines migrated their green-screen-based systems, which couldn’t integrate with other systems and services to SOA, which covers the product suite for reservations, inventory and passenger check-in. The mirgration as the equivalent to a “heart transplant,” according to Shama Patel, business program manager of the SOA effort for United and Lufthansa. Eventually, Lufthansa and United plan to roll out a common platform that can also be used by other members of the Star Alliance. “The modernization project will impact 20,000 people in 350 locations in a three- to four-year time frame, and it will touch 20 company divisions.” (Source: CIO)

To increase logistics efficiency. A $5 billion logistics and trucking company ought to know plenty about economies of scale. It may cost $1,000 to ship one refrigerator from New York to Los Angeles, but cost a penny if it’s intelligently bundled with another shipment. Con-Way has been applying this multiplier effect to its information technology infrastructure, with significant paybacks over the long run. Con-Way set out a number of years ago not knowing how much services would be reused — but this was very much their design goal. “When we built services at that point in time, we built every piece of functionality as a reusable piece of code,” said Shibashis Mukherjee, Con-Way lead enterprise architect. “We had no idea whether it was going to be reused or not.” However, reuse across the company’s various business lines took off. “You don’t generally see the benefits in the first project you do. We also had executive management buy-in and we had a long-term vision. So as project after project is done, our development time was cut as we reused components built by the previous projects.” (Source: SearchSOA)

To speed up mergers and integrations. Dematic, a $1.5-billion global logistics company, recently replaced its core IT architecture in 72 days with a platform built on SOA and Information Technology Infrastructure Library (ITIL). Sound implausible? The company’s IT department was given a 90-day mandate to integrate its systems when Siemens sold the company to Germany-based Triton in late 2006. CIO Allan Davies said the company saved money by abolishing legacy systems which Davis said chewed up a considerable amount of cash. “Our expense management system, hosted with Siemens, required four daysa month of manual data entry into our enterprise management system,” he said. “SOA lets us basically plug and play. I know how painful legacy systems are and this means we don’t need to mess around because we need to be very responsive. We have a lot of feeds from our systems that have to be managed and we tended to write a lot of interfaces — we have eliminated the legacy systems.” (Source: CIO)

To cut IT costs through reuse. At Delta Airlines, SOA is cutting the cost of ownership by half for its various applications and systems. “Reuse is one of the big drivers for our SOA environment,” said Bret Martin, principal enterprise architecture for Delta Technology Inc. Delta’s SOA, for example, is reusing the same customer and operational data across a range of systems, from the Delta.com Website to ticketing kiosks to ticketing counters and gate systems. “It allows check-in to happen in a uniform way,” he explained. Another way services are being leveraged are by exposing services to vendors and partners, such as American Express or operations companies.

To help plan regional natural resource consumption. The Southwest Florida Water Management District originally managed data and transactions on a mainframe system, but in planning for is Water Management Information System (WMIS) — designed to automate and streamline the paper- and time-intensive well-construction permitting process — it was decided to move off the mainframe and onto a distributed system developed to support SOA approaches. The new system will better automate the permit application and approval process, while providing Web access and supporting geospatial data. The District reports that 86% of its 17,000 annual permits for well construction are now handled electronically, and 35% of Web services applications are reusable across the agency. (Source: Redmond Developer News)

To prevent black holes from tearing the space-time fabric. It doesn’t look like CERN’s Large Hadron Collider (LHC) has produced any black holes (unless you count the stock markets), but it’s nice to know that the European Organization for Nuclear Research is employing state-of-the-art technology to keep things in check in case a black hole does accidentally form. CERN is employing SOA-based software to monitor and manage potential Large Hadron Collider (LHC) emergencies. CERN employs an enterprise service bus to form the communications backbone of its Technical Infrastructure Monitoring (TIM) system, which collects, evaluates, stores and distributes data about 2.1 million items every day from 150 different systems and 60,000 different measuring points. The ESB unites disparate data and systems without hampering CERN’s research efforts by underpinning their ability to get systems back and up running as quickly as possible in the event of a fault. “It’s all about speed of detection and action - we can stop and restart systems in a matter of seconds with the software,” said Eric Lienard, CERN technical infrastructure manager. (Source: ITPro)

December 24th, 2008

SOA unplugged: what readers had to say about SOA in 2008

Posted by Joe McKendrick @ 10:06 am

Categories: Business ROI, Case Studies, Enterprise Architecture, Event processing, General, Management, SOA Events, Standards Watch, Vendor Watch, Web 2.0-Enterprise 2.0, Web Services

Tags: SOA, Service-Oriented Architecture (SOA), Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

I’d like to take this opportunity to thank all you readers out there for visiting this blogsite over the past year, and I appreciate all the feedback and commentary received through the TalkBack feature. A lot of compelling and frank views were shared and discussed.

I thought it would be interesting to repost some of the comments that some of you provided over the past year as they relate to SOA issues discussed.

On the relationship between SOA and Complex Event Processing, James Taylor said: “I think one of the powerful effects of automating and managing operational decisions is that it helps link SOA, EDA/CEP and BPM. Decision-making logic must be shared effectively across these different approaches and that means that decisions must be identified and managed as first class objects.”

On the relationship between SOA and business process management (BPM), Chris commented on the challenge of achieving stateless and loosely coupled services: “We have been looking for ever more flexible ways of creating components. We have been looking for as little coupling as possible between our components because that makes it easier to localize changes and reduce unwanted side effects. We have been looking to increase cohesion because keeping ‘things’ that belong together together gives us an easier way to manage the thing. We create deployment mechanisms that allow us to scale our components by adding new instances. We create deployment mechanisms that keep the functional behavior away from the operational details. But still we have trouble with the right level of abstraction to get value out of the components. That’s partially because we have relative complexity in our businesses, we are encouraged to design for the ‘pretty case,’ i.e. the case where stuff doesn’t break and we don’t deal with timing and ambiguity well… So the  hope of having ‘process stateless’ services is a longshot. Nice to strive towards, but not easily achievable.”

On SOA and economic conditions, Kirstan said: “If the economy is turning down, then free and open source tools to help build and run an SOA make more and more sense. Open source tools like XAware for data services, ActiveEndpoints for orchestration, Mule or ServiceMix for ESB, are quite mature. Aside from saving money in a downturn, open source tools make even greater sense for ‘project level SOA,’ where a company can build an SOA in the small before committing to an expensive enterprise-level SOA.

On SOA even makes the US Marines nervous, Jabailo1 said: “SOA is the only way to go. Unfortunately for large organizations which have grown fat and stupid on “application servers”, outsourcing and management pay increase, it also takes finesse, intelligence, hard work and most of all — that bugaboo word — programming! Yeah, all those guys you laid off seven years ago? Those are the guys you want. All those guys you kept and made into “technology managers” and gave 14% per annum pay rises while they shipped the budget to Bangalore? Those are the guys you want to get rid of…”

On the “emergence” of event-driven architecture (EDA), Rob Eamon said: Analyst groups “have been promoting the notion of the event-driven enterprise for over a decade. In my opinion, EDA and SOA are orthogonal notions. One doesn’t subsume the other. A business or enterprise architecture is likely to adopt, or should adopt, principles from both approaches.”

On reuse being a primary goal of SOA efforts, Eelcoh said: “SOA should not be about reuse at all. If there is, and will be, only one consumer, there might be two different situations: 1 - The providing department is wrongfully made responsible for the process (that might happen). In this case, it does not make a lot of sense to have that department provide the service, but unless the right department is found, the alternatives aren’t that good either. 2 - It just happens to be that there aren’t that many consumers. In the second case, there should be no question about who should provide the service. The alternative would be that the consumer one way or another replicates the process. That is what SOA prevents. So one could say it is not about reuse, it helps to reduce. In the end, in my opinion, SOA is about isolating change, but I doubt whether the vendors understand that.”

On ‘how to tell it’s not SOA,’ Tonymcs flipped the scenario with ‘how to tell if it IS SOA: “If it it doesn’t work, it’s SOA…. If its a 3 hour lecture with no information content, it’s SOA…  If it’s recommended by non-tech people, then it’s SOA… If it’s as useful as art criticism, it’s SOA… If it’s slow, it’s SOA…. If it’s spaghetti Javascript, then it’s probably SOA….. If it’s a bunch of services you’ll never use, it’s SOA…. If it’s a category looking for a reason to exist, it’s SOA.”

On the evolution of computer science to “service as a science,” Storm14k said: “Sometimes I laugh at these paragraphs that basically say nothing or restate common place scenarios with a lot of business buzzwords: ‘Service systems are complex systems that dynamically configure access to resources (people, organizations, technology and information) to interact with other service systems and mutually create and capture value….’  Now ask one of these guys to build a concrete example of this and then the truth comes out. Its a load of crap and once they do figure it out they’ll turn to the nearest guy with a CS degree to implement it anyway.”

On selling SOA to the business, Jean-Jacques Dubray said: “There is nothing ethereal about SOA. The very reason why most people think SOA is ethereal is precisely because most developers, architects, business analysts were never presented with a comprehensive view of SOA. Most people claimed they were doing SOA 20 or 30 years ago and that WS-* is not SOA. The reality is quite different, concepts such as bidirectional interfaces (WSDL), semantically accessible data structures (XML), orchestration (BPEL), forward compatible versioning (XML, XSD, WSDL)… are all brand new ideas that never existed before 1998 or so. SOA is only ethereal if you care to ignore these concepts.”

On SOA as a strategy for times of economic turbulence, Robert Morschel said: “Not convinced. The biggest problem that SOA is trying to solve is the total cost of ownership and the lack of agility caused by years of tack-on systems development. That’s where the big money saving potential is, not making business processes more efficient. I agree SOA introduction needs to be iterative, but against the backdrop of an already persuaded business who are sick and tired of paying 60-80% of their IT budget just on maintenance costs not new development.”

On whether SOA may be a faddish — and even sloppy — approach to management, Donald said: “Those who say that SOA is just a fad or a buzzword do not really understand all the necessary concepts, governance, new application development and execution paradigm, and technologies needed to implement the simple concept of SOA. For a decade people had hoped that PKI [Public Key Infrastructure] would just go away because implementation is too complex for most people to want to understand. But today PKI is the only name in town for various identity purposes. SOA will not go away because it is the only name in town to shorten time to market and to save millions of dollars.”

On whether SOA is more about integration or architecture, GGruber66 said: “SOA=Integration is a very narrow view. And most people who are looking at it that way aren’t just missing 1/2 the point of what SOA can do, they’re also missing 1/2 the point on integration. SOA is not a panacea for integration. Passing data is easy, but if you’re not moving to the business process aspects of ‘what do I do with the data now that I have it’ you’ve missed the boat. And SOA doesn’t do that. Frankly if all you want is integration, just use a service like Boomi. SOA’s real impact is felt when people expose services to others and allow people to create new capabilities and new products or services (I don’t mean Web services). And when they take their eye off the fact that as you state SOA is an architecture, the services that they create probably won’t meet the requirements for reliability and availability that businesses large and small demand. This is what happens when people fall in love with a buzzword and don’t try to understand the business value that it’s supposed to create. The SOA projects that fail most are the ones that start with SOA as the answer before anyone really thinks about what the question was.”

October 28th, 2008

Situational awareness, thanks to SOA-driven dashboard

Posted by Joe McKendrick @ 7:17 pm

Categories: Business ROI, Event processing, General, Links, Management, Vendor Watch

Tags: Dashboard, SOA, Service-Oriented Architecture (SOA), Enterprise Software, Web Services, Software, Joe McKendrick

Everyone knows what a dashboard can do, and yes, that’s SOA behind your dashboard.  So, therefore, for dashboard presentations that really dig into what’s happening across the enterprise, the business should be more than willing to support SOA.

Rich Seeley recently posted some perspectives on Oracle’s Business Process Management 10g Release 3, a business intelligence dashboard that enables business users to customize their view of data coming from Oracle’s SOA-related technologies, including Complex Event Processing, ESB and BPEL.

The dashboard puts capabilities right in front of decision makers’ eyes. They may not entirely get SOA, but they clearly see the analysis of this quarter’s sales results and inventory levels displayed in front of them.

Roy Schulte, vice president of Gartner, is quoted as observing that the executive dashboard may be among the keys to bridging SOA and Business Process Management. “Business users may not understand SOA, BPM, CEP or XTP [eXtreme Transaction Processing], but they know what they want to see on their dashboard and they may be willing to fund back end architecture and development projects to get more information faster.”

Of course, if it’s nothing but bad news being displayed on the dashboard, the executive may just want to throw the whole thing out the window.

October 28th, 2008

Why 'Event Driven Architecture' is more than 'Complex Event Processing'

Posted by Joe McKendrick @ 6:49 pm

Categories: Enterprise Architecture, Event processing, General, Links

Tags: Event, EDA, CEP, Service-Oriented Architecture (SOA), Productivity, Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

Today’s mantra:

“EDA is an architecture; CEP is a technique…”

Jack van Hoof, who has written extensively on all things EDA, says there have been one too many instances in which the terms EDA (Event Driven Architecture) and CEP (Complex Event Processing) have been used almost interchangeably. He provides a clarification:

“CEP is a way of processing messages (fair enough to name these messages “events”)…. But EDA is about how business events drive the overall architecture of the IT-systems and it is about how these events should be modeled. EDA it is not primarily about the ability to process and correlate streams of thousands of messages per second…”

Just as SOA addresses the architectural aspect of service orientation for the business, EDA addresses the architectural aspect of how events should be processed for the business, Jack says. “CEP is just one among these aspects.” Please, he adds, EDA “EDA does not only deal with complex events (correlations) but also with simple events.”

“So CEP is not EDA, EDA is more than CEP. Promoting CEP as being EDA is far too simple. And yet that is what is happening in the current IT space.”

October 27th, 2008

First 2009 SOA predictions arrive; see fading of the hype

Posted by Joe McKendrick @ 9:13 am

Categories: Enterprise Architecture, Event processing, General, Links, Management

Tags: SOA, Dave Linthicum, Service-Oriented Architecture (SOA), Web Services, Middleware, Enterprise Software, Software, Joe McKendrick

Dave Linthicum has beat everybody to the punch and issued his predictions for the year ahead in service oriented architecture.

These are good predictions, my thoughts added:

1. The interest in cloud computing will drive many enterprises toward SOA.

Agreed. Cloud computing and software-as-a-service (SaaS) will make SOA real for many business users. In the same vein, user-driven, self-service functionality, such as mashups, will also put SOA more front and center to the business.

2. The explosion in PaaS (platform-as-a-service) will leave many enterprise architects and CIOs scratching their heads.

Agreed, with additional thoughts. PaaS-style offerings may take some of the technical headaches out of IT infrastructure, and make things a lot cheaper. Integration-as-a-Service will offer a compelling alternative for SOA efforts as well.

3. The economy will recover, but most enterprises out there will focus on cost reduction.

Agreed. But the focus on IT cost reduction has never let up since the 2001 downturn. This is where SOA can potentially pack a punch.

4. There will be a larger focus on inter-domain SOA technology, or highly scalable and secure middleware technology that will provide scalable service and information access between the instances of SOAs within the enterprise, and perhaps intercompany as well.

Agreed. Most companies now have islands of SOA; expect more efforts to bring it together.

5. Jig will be up for poor SOA governance solutions out there.

Hmm. There are too many spaghetti oriented architectures and JBOWS architectures out there — in other words, enterprises with NO governance solutions. We’re not likely to see a letting up on these, or a move to well-governed SOA projects en masse any time soon. We’re still in the early stages of this continuum, giving vendors some time to get things right.

6. Most failed SOA projects will be traced to unqualified SOA architects.

Hmm. Good SOA architects are in short supply; the rest will continue to muddle along, learning by the seat of their pants.

7. SOA the buzzword will become a bit less relevant and will begin to morph with concepts, such as enterprise architecture and cloud computing.

Agreed. Vendors will start to move on to new things. Other possibilities: Enterprise 2.0, Event Driven Architecture.

October 12th, 2008

Response to financial crisis may mean more IT work ahead

Posted by Joe McKendrick @ 9:05 am

Categories: Business ROI, Event processing, General, Links, Management

Tags: Financial, Information Technology, Industry, Enterprise Architecture, Strategy, Regulations, Data Management, Financial Accounting, Management, Government

Governments all across the world are responding to the current financial crisis with a range of nationalization initiatives, and increased support of various parts of the financial services industry through guarantees and pumping in increased liquidity.

You know what it means when there’s more government involvement with enterprises. Lots of regulations. Lots and lots of regulations. New channels of accountability. In addition, you can almost see the handwriting on the wall, especially here in the US, the epicenter of the crisis: there will be some kind of new legislation, a la Sarbanes Oxley, requiring greater transparency and regulation of mortgage markets and associated financial instruments. “Never again will we allow ourselves to get into this mess” will be the urgent rallying cry coming from all sectors.

A couple of weeks back, ZDNet colleague Jason Perlow discussed the impending IT shift that may accelerate as a result of the current crisis. I agree with Jason that we’ll see a lot more infrastructure consolidation, with SOA, virtualization, and cloud computing leading the way.

Ron Tolido, CTO for CapGemini in the Netherlands region, recently raised some interesting points about degree of this impact, agreeing that the fallout from all the financial turmoil raging around us will include the need for more and better information technology.

As he put it (also quoting some points raised in this blogsite about companies requiring more enterprise architecture skills to help streamline and consolidate):

“It sounds utterly cynical, but we should not be surprised if the current misery leads to a pile of new IT work. The government is clearly back at the wheel in many places and the public opinion just won’t tolerate even the slightest suggestion of misconduct in business. Rules and regulation will be tighter than ever. It means more risk management, more reporting and in general a call for total transparency. All of these areas obviously depend on enabling information systems.”

Rough waters will also result in more mergers, acquisitions, and spin-offs in the months and years ahead. Just this weekend, there’s been talk of merger talks between General Motors and Chrysler. In other words, a lot of integration work between massive IT infrastructures. As Tolido put it, “this will require extensive support through enterprise architecture, standardization and carefully crafted integration solutions.”

When the IT industry was hit hard in 2001 with the post dot-bomb crash, one of the drivers that helped bring the industry back was a slew of new regulations and mandates, such as SarbOx. Companies suddenly had a desperate need for data management tools and platforms that helped them comply.

From a private sector perspective, enterprises across all sectors are going to want to be even better able to analyze and predict changes and shifts in their markets. Could more effective information technology have softened the blow to financial services companies from the subprime mortgage debacle? Perhaps, in combination with sound business sense. Competing on analytics is taking on an even greater urgency than before; the ability to connect, leverage, and interact with networks of customer and partners will separate the losers from the winners.

Busy times may lie ahead for the IT industry, if the industry steps up with solutions directed at increasing the transparency and agility of the enterprise.

September 18th, 2008

'Out of your brain and onto a hard disk': Microsoft's Oslo SOA push

Posted by Joe McKendrick @ 12:54 pm

Categories: Event processing, General, Links, Vendor Watch

Tags: Hard Drive, Oslo, Microsoft Corp., Service-Oriented Architecture (SOA), Web Services, Web 2.0, Middleware, Enterprise Software, Software, Internet

For Microsoft, Oslo not only represents its future intentions to bring SOA sensibilities to the commodity computing space, but also to make it easier for non-tech types (gulp) to build services and applications.

Where exactly did Oslo come from, and where does Microsoft intend to take it? For those that are wondering what Oslo is all about, Darryl Taft put together this overview that takes a deep dive into the intricacies of Oslo.

Oslo is not a single product, but a capability that is being built into future products, such as BizTalk Server, to enable the rapid building and deployment of composite applications — which supposedly makes it easier to build using SOA principles.

Microsoft’s original purpose of Oslo was to make software development easier by enabling people (and not necessarily just developers) to create applications from models or diagrams. Microsoft’s Don Box is quoted as stating that the goal of Oslo is “We’re trying to make it simple to get an idea out of your brain and onto a hard disk.”

Part of the Oslo effort includes a new language, code-named “D.” Why D? What’s wrong with C# — the .NET core language that has finally started to catch on with developers? The D language was developed as a declarative programming language that delivered a visual tool for creating models along with a repository to store the models and metadata.

Web 2.0 is making is easier for end users to assemble their own applications, and this is clearly the direction SOA needs to take. The agility to be gained from quickly assembling or de-assembling services to address ever-changing business requirements would be lost if the business has to keep going to their already overburdened IT departments to make the adjustments.Microsoft seems to understand that the momentum is moving toward business end-user empowerment — that’s how they got to be such a big company in the first place. But will Oslo and D make SOA as simple and easy to grasp as Web 2.0? Microsoft seems to acknowledge that the D language will be too much for many people to learn, and is harder than Excel or Visual Basic.

ZDNet’s Mary-Jo Foley also provided some preliminary details on Oslo a couple of months back.

Rob Helm (Directions on Microsoft) also published a good overview (PDF) on Microsoft’s Oslo plans.

September 15th, 2008

'Business event processing' now in the spotlight

Posted by Joe McKendrick @ 3:27 pm

Categories: Event processing, General, Links, Management

Tags: Event, IBM Corp., Business Event, BEP, Service-Oriented Architecture (SOA), Supply Chain Management (SCM), Branding, Web 2.0, Web Services, Middleware

Last week, in one of its classic 300-product mega-announcements, IBM unveiled what it calls its business event processing line, “representing a unification of several IBM global initiatives, including Service Oriented Architecture (SOA), Information on Demand, virtualization and Web 2.0.”

Is ‘business event processing’ too much to swallow?

Business event processing is one of those terms that really sounds cool and a must-have. But what is BEP? Is BEP a good bet?

My esteemed colleague Tony Baer, who has been following this space for some time, congratulated IBM for having the common sense to recognize what it already had, and to start positioning products under the BEP umbrella.

However, he also cautioned that “event processing is not something you buy with a single shrink-wrapped application.” (Hmm… sounds like SOA…)

Tony explains that event-processing “technologies are diverse, and so are the use cases. For starters, lets ask, just what is event processing? Some call it event processing to refer to something generic; there is also complex event processing where you parse intricate sequences of events or occurrences that could only be detectable by machine; and then there is business event processing, which is the idea that we’re not just talking about events in the abstract.”

Tony says BEP takes in a huge range of disciplines, including  high performance, low latency, high throughout, structured and/or unstructured text or rich media data, and detection of highly complex event patterns requiring complex rules processing, among other things. In other words, a lot of solutions from a lot of different areas of IT, and some of them non-trivial implementations:

“We wonder whether there is such a thing an event, business event, or complex event processing market per se. We have our doubts, not because the category is so difficult to bound, but because event processing is not an end in itself or discrete market. Instead, event processing is part of business solutions ranging from telco provisioning to capital markets trading, check or credit clearing, supply chain optimization, law enforcement and homeland security, and so on.”

IBM seems intent on offering some sort of point solutions that would at least get some companies started on the road to BEP. Big Blue says the purpose of BEP technology is to “automatically track, analyze and respond to changing business conditions — whether the events that trigger them are planned or unplanned.”

Why is this important? IBM cites the explosion of business information from email, voicemail and new sources such as RFID, and estimates that there are now up to 72 quadrillion unique business events generated worldwide each day. “The ability to process these information streams quickly and accurately is one of the most complex tasks facing businesses today, requiring massive computing power and the most advanced software and hardware available.”

Gartner VP Roy Schulte recently estimated that a typical large company may have up to a trillion events per day somewhere within its networks.”

The question is, of course, which of these billions and trillions of events need to be captured and automated. That’s going to take a while to sort out.

May 23rd, 2008

Sense and respond -- should humans be part of complex event processing?

Posted by Joe McKendrick @ 9:02 am

Categories: Business ROI, Event processing, General, Links, Management, SOA Events

Tags: Human, Application, Event, SOA, EDA, Service-Oriented Architecture (SOA), Web Services, Middleware, Enterprise Software, Software

This week, a report out of the Financial Times (cited here in the LA Times) said that a bug used in the computer models used by analysts at Moody’s Investors Service caused Moody’s to award “incorrect triple-A ratings to billions of dollars worth of a type of complex debt product.” The report alleges that the errors were not corrected even after the code was fixed a year ago.

The implication is that millions, if not hundreds of millions, of dollars in investment decisions may have been made on faulty data generated by the system. If you’ve been following the financial markets lately, it was not a good time for that.

This once again raises the question of how risky is it to take humans out of the loop of event processing? Of course, humans are fallible, and usually create the messes we then look to technology to fix.

Dr. David Luckham, considered the “father of complex event processing,” provided a sterling example in a recent ebizQ panel discussion of a very complex event that needed a lot of processing: World War I.

World War I — which was sparked by a chain events from an assassination in the Balkans, and was followed by a chain of events that exploded into World War II — was probably too complex for any human or machine to process the various inputs and outcomes.

Read the rest of this entry »

May 11th, 2008

Is anyone ready to process a trillion events per day?

Posted by Joe McKendrick @ 12:04 pm

Categories: Business ROI, Event processing, General, Links, SOA Events, SOA Surveys and Research, Web Services

Tags: Event, Supply Chain Management (SCM), Sales Strategy, Financial Services, Enterprise Software, Software, Sales, Joe McKendrick

What is event processing and how could it make a difference? Here is one example of where it could take us: Scientists are talking about the possibility of an in-body network that could detect heart attacks or diabetic collapses and alert emergency services.

A typical company deals with millions, if not billions, of events in a single day, and most are still handled manually, by people

Interesting idea. But then the article goes on to suggest that the same technology could be used to monitor whether patients are taking their medicines. “A pill dispenser would send an automatic reminder and, if the pills were not taken within a certain time, an alarm would sound and a message would be sent to the patient’s family or caregivers.”

So we have the need-to-have, lifesaving value of event processing, and the intrusive, nagging aspect. Now, apply this kind of scenario with a business. That is, if the ongoing health of the business is monitored and maintained with intelligent technology, serious issues could be headed off either by alerting decision makers before things happened, or even in an automated fashion. But is it possible to have too much processing of events that aren’t as critical, and merely drive staff nuts?

Of course, the challenge is that businesses require more than taking a few pills each day to keep problems in check. And note that the pill-nagging system still requires human intervention at the end of the process. A typical company deals with millions, if not billions, of events in a single day, and most, if not all, of these events are still handled manually, by people. It’s obviously not an efficient and competitive way to do things, and the more you can automate, the better. But how badly is it needed?

Gartner VP Roy Schulte recently made a compelling case for increased complex event processing in an online presentation delivered as part of ebizQ’s recent Event Processing virtual conference (registration required). “At any one second, a large company has on its network anywhere from 10,000 to 10 billion business events,” Roy explained. “At the low end, that’s almost a billion events per day — at the high end, that’s almost a trillion events per day.”

The question is, which of these billions of events need to be captured and automated? Read the rest of this entry »