What’s Really Happening in the World of CEP
There is quite a lot is happening in the world of complex event processing. Interestingly enough, the people and the companies advancing processing complex events are not calling what they are doing “CEP the Buzzword” we read about in the press or that a handful of “pioneers” claim make them modern-day CEP “experts”.
The companies processing complex events are using advanced statistical algorithms for detection. Look into technologies like Google Googles, where Google uses image feature extraction with Bayes-driven statistical algorithms toward a futuristic capability of snap-and-send image recognition and context retrieval all over the net in real time. Of course, single image recognition by itself is not event processing. Complex event processing occurs in the later processing stages, when multiple images are used to abstract current events or predict future ones.
Looking for real “pioneering” CEP versus the rehash of the same old financial trading systems repackaged as new ones?
Then look no further than where Google is going with voice-to-voice language translation over the network, where you talk into your phone and your voice is translated to text on the backend, and then Google translates the text to the language of your choice (51 at the moment and headed to 100 according to Google). This is complex event processing. Google’s VP of Search called this “the secret of unlocking the web.” You may recall that a number of the first “true CEP” applications (long before the phrase CEP was coined), were speech processing systems. Early blackboard systems were designed for this very difficult and complex problem.
For more traditional detection-oriented complex event processing, look at what is happening at CERN with the Large Hadron Collider (LHC). CERN fired it up and it is starting to get interesting! Detecting the fundamental particles that make up our universe, with over 10,000 scientists working on detection and analysis in (perhaps) the most expensive and exciting complex event detection engines ever created.
That is real complex event processing. Don’t be fooled by simple stream processing engines calling themselves complex detection engines or CEP companies.
Therefore, please don’t be mislead by the software marketing of the “self licking ice cream cone companies” of financial services (who have diluted the meaning of CEP to mean, “whatever we do in financial services with a stream engine”. It’s just the same-ole same-ole from the same old folks, detecting very little of interest to anyone.
On the other hand, there is an exciting world of complex event processing out there, and I have only scratched the surface in this post.
“CEP the financial services hype” is boring and mundane compared to the very exciting world of processing complex events in the real world.
Note: I originally discussed this at Marc Adler’s blog, Magmasystems.
Filed under: Advanced Event Processing, Analytics, CEP News and Events, Complex Event Processing, Cyber-Trading Technologies, Event Processing, Financial Services, Predictive Business, Use Cases
[...] Seems like not a lot has changed in the last year in the world of Complex Event Processing - and I agree with a lot of what Tim Bass has to say in this post. [...]
Tim,
I agree with a lot of what you have to say here. More at http://colinclarkeventprocessing.com/?p=296
How about some use cases from those that are doing complex event processing and not calling it CEP? Or some use cases that the current ESP products (which call themselves CEP) have difficulty with?
Glad to see you blogging again. In my mind, the type of processing done by autonomous cars, drones and miniature robots is far more interesting example of processing events and data streams. If one were to ask the people doing that type of R&D, they would say event processing is just one part of the larger problem.
It’s really disappointing and sad current products are driven more by hype and marketechture than facts and solid research. my bias 2 bits.
The next time I get to work on an international multi-billion dollar project involving massive hardware infrastructure for event detection whose 15 Petabytes of annually generated data serves as the basis for analysis at the bleeding edge of modern theoretical physics, I shall remember this post and be thankful that I can move on from the boring and mundane world I live in today.
Hey Charles,
You don’t have to work for CERN to enjoy very cool complex detection problems or to have access to the software CERN uses.
These problems exist just about everywhere, from security to network management, from search-and-retrieval to classification, to spam detection to state-of-the-art fraud detection.
The world has moved far beyond rule-based approaches and this has become the discriminator between companies that are moving forward and companies inching along.
OBTW, according to CERN, most of the software used at LHC is free open source, written in every day pattern recognition languages like Perl and Python (as I recall).
We tried at least two of the so-called CEP engines for processing security events on two busy web servers, and dumped both of them quickly and ended up using regular expressions and standard unix/linux scripting languages for rule-processing. We don’t need no “stinkin’ silly GUIs” to process streaming events, LOL. We need regex and a bit of logic.
Cheers.
I am wondering how much classification, naming, etc. we actually need. After all, naming something allows us to reference the concept (or thing) with a common context. If we don’t have a common context then names get in the way!
So, CEP. Having read many learned articles, books, papers on the subject I don’t really know what any of it actually means except that most work pushes a self aggrandizing point of view. Either reputation or software.
So, I look at the problems at hand in whatever I am doing, attempt to reason through them, talk to a few experts, maybe read the online or other published materials and formulate a plan or approach for my problem space.
Sometimes that will result in looking closely at software, sometimes that will result in, “Have you gone out of your cotton picking mind, that problem is intractable because?”. Sometimes it results in simply extending some existing semantics. It so much depends on the problem space and what you are doing in it.
Of critical importance is the Event Distribution Network - how the events get distributed/routed to where they need to go for proper situational awareness or other handling. Whether we have autonomous actions arising from multiple “implications” of the same event. How the events fit within a pattern of behavior… There are so many cases/types/ways of thinking about this that we can’t take a universal one size fits all approach.
Multiple petabytes of small events that have to be responded to in concert in real time is at one extreme of challenge. Distribution to all interested parties of a change in bus schedule so that the interested parties can make proper decisions about sequencing traffic lights, allocating parking, handling hi-occupancy vehicle lanes is at quite the other end of the spectrum.
So anywhere from real time, situational awareness to distribution of less frequent, slower response needed business events all have to be thought about and appropriate decisions made. So don’t just go and buy a tool because it’s cool. get what you know you need, realizing that you don’t have the full scope figured. Ask about likely patterns of use, and realize you will need multiple ways of managing your event delivery networks.