Cyberstrategics

It was almost three years ago when I was introduced to the buzzword “complex event processing” by good friends and colleagues at TIBCO Software.  At TIBCO (before and during my time there) we always discussed how “real-time event processing” would be a technology to address “the needle in the haystack” detection problems in the big “event cloud” and we often discussed many of the technologies that would be required moving forward.  TIBCO, true to their visionary leadership in enterprise software, has made two strong related acquisitions since that time (Spotfire and Insightful) and have been working to integrate these technologies.   This post, however, is not about TIBCO, Spotfire or S-Plus; although I keep promising everyone that a post on this integration is coming.

This post is about how far CEP has fallen behind in the past three years.  Yes, we continue to see CEP vendors and their favorite analysts toot their own horn; and yes we continue to see some purchases of stream processing software, packaged as CEP, on the trading side of financial institutions.   However, this minimal progress is not really related to progressing the state-of-the-art of processing complex events.  What we are seeing, basically, are folks in financial services enjoying a bit of new found freedom to process market data and having great fun calling these stream processing applications “complex”.  The reality is that most users do not derive much benefit from CEP beyond a niche market in financial services.

On the bright side, during the past three years we have seen some truly remarkable advances in other technologies.  Open source applications, especially open source applications (of all types) based on Linux, Apache, MySQL and PHP (LAMP) continue to grow at an amazingly rate.  These advances are user-driven, not marketing driven; which brings me to the main point of this post today.

The beauty of information technology is that it is generally user-driven and the users can change their mind and adopt a new technology very quickly.  I believe some of you can remember the Internet in the  early 1990s when we were thrilled to have TELNET, Gopher and WAIS to access network systems and retrieve data.  Then, when the Web and HTTP came along, we were even more thrilled (most of us, that is!).  These were killer apps and these killer apps did not need any marketing hype or dog-eat-dog competition and amnesty programs over a few bones in a narrow market segment.  These killer apps touched everyone’s lives. These applications brought people together and did not tear them apart.

CEP, on the other hand, has gone the direction of a “new algo trading engine” and the focus from processing complex events, detection of complex events, and identifying causes of complex events, has evolved to a marketing buzzword for a very small, aggressive niche market.   In fact, the aggressive marketing and hype-generation has become so buzzword driven in capital markets, that I now receive some form of hate mail almost weekly, and this hate mail is always from someone who has a career related to market data and trading applications.    Conversely, I receive praise, almost weekly, from people outside of capital markets; so there we have it.

The results are pretty amazingly simple and we can easily see what is really happening.    CEP, as a concept for actually detecting and analyzing complex events, as envisioned, is falling behind the power curve.  Users outside of a small niche group of folks interesting in algorithmic trading and order routing, all suitable for stream-based rules-processing, have not materialized.    In fact, nearly every Google alert I received on “complex event processing” today was marketing and analyst reports based on “CEP” in a trading, or related, stream-processing application.  The only other “CEP” alerts from Google were my blogs posts, which are seemingly increasingly offensive to the trading community, and a post from TIBCO, which was a repeat of what we were saying three years ago.

In other words, the CEP community has become very narrow; and it will be very difficult for such a narrow community to have much impact if it does not find user acceptance outside of the niche trading space.   Yes, we do hear of some progress behind-the-scenes in other areas, but the airware are deafeningly silent for the most part.  CEP, the concept for actually detecting and analyzing complex events, is falling behind the power curve.    There are much more exciting technologies on the horizon today.

Filed under: CEP News and Events, Complex Event Processing, Event Processing, Event Stream Processing, Fraud Detection, TIBCO