A Complex Event = Sum (Events) + Situational Knowledge
Sometimes we read some opinions about CEP where folks opine that ”complex event processing” is really about processing “complex events” and not about “complex” “event processing”. The truth be told, processing “complex events” requires “complex” “event processing” so there is really no difference between the two ways of expressing CEP.
You can not process complex events in some very simple way and expect to get accurate results. You need knowledge, represented by one or more situational models, to process complex events.
Some folks, like to say that a “complex event” is simply an event which is an aggregation of two more more event objects. If you follow this (flawed) logic, then counting integers is complex event processing; because 1 plus 1 is 2, and 2 is an aggregation of 1 and 1, so 2 is a complex event (not!).
Since we know that counting is not a complex processing operation, some folks would say that you can process complex events with very simple operations because you are processing complex events, in this example case adding 1 to the previous number (counting), enriching an event object.
This is simply nonsense.
The logic flaw is that the basic definition of a “complex event” (used by many people) is wrong. A complex event is not simply an event object with two more more events as sub-components.
A complex event is when two event objects are combined (processed) to form a complex object with a higher degree of inference, or situational knowledge. One plus one equals more than two in complex event processing, because the combination of event objects requires knowledge (e.g. a situational model).
A Complex Event = Sum (EventsObjects) + Situational Knowledge
Let there be no mistake about it. Complex event processing is the complex processing of complex events. You cannot accurately process complex events with simple event processing models.
The simple processing of complex events is not CEP, it is simple event processing (event track-and-trace, simple event object enrichment, simple event object aggregation, and so forth).
Note: In this post, I use Complex Event = Sum (EventsObjects) + Situational Knowledge as an abstraction only. It might be more accurate to say Complex Event = F(EventsObjects) + Situational Knowledge. where F(EventsObjects) represents some ”complex” operation, for example, a Bayesian Classifer, an Artificial Neural Network, or sophisticated rule-based expert logic.
Filed under: Advanced Event Processing, Analytics, CEP Terminology, CEP Tutorials, Complex Event, Complex Event Processing, Detection Theory, Event Processing, Event Processing Modelling, Event Stream Processing, False Positives and Negatives, Modelling and Simulation, Simple Event Processing, Situation Models
Hi Tim,
I liked “Complex event processing is the complex processing of complex events” very much.
There is lot of confusion about CEP and you are doing good job here by making things clearer. By simply building 1+1=2events and calling it as “complex events” would not help unless there is a context attached to it. The real value is in building situational models and build complex events inside that context. And anything unusual outside this model needs attention.
In practice I find it pretty tough to convince the relevance of CEP to technical staff and the problem with business is that they don’t understand CEP well.
-Hari
Hi Hari,
Thanks for visiting and for the comment.
I think I am having some effect, because David Luckham just demoted The CEP Blog from being on the top of his CEP blog list, to the bottom. I’ve been punished, I assume, for not towing the party line.
Yours sincerely, Tim
that just means you’re doing something right and expressing the same concerns many users have, but don’t say aloud. It takes courage to speak out, especially when the tide is moving in the opposite direction.
Hi Peter,
Great to see you here on The CEP Blog.
Yes, I recall almost 10 years ago when I suggested using the concepts from multi-sensor data fusion (MSDF) for network intrusion detection, and I was “ripped apart” by a team (CIDF) getting DARPA research dollars et al.
Now, sensor fusion is the leading model for IDS, and the folks leading the opposition back then have all but disappeared from the scene, or claimed they invented the fusion approach!
Really, I think the CIDF opposition then, now actually claim “they invented” the same approach they opposed when I introduced the sensor fusion model for network and computer security.
As you can see, I have been in this uncomfortable situation before, and know how bumpy the road gets; but I also know that being right always wins in the end; and the folks who claim “foul” now will eventually claim what I am advocating now was “their idea” all along. That is when we know we were right, because folks will claim they were thinking along these lines all along!! Some will even go for patents if they have a deep pocket legal budget.
We are already starting to see this, as a few core EPTS folks now say they support sensor fusion, analytics, etc, as core to CEP; when only a two years ago they were talking only streaming database technology.
Deja vu all over again
Cheers.
Yours sincerely, Tim
Tim,
I believe that David’s site is rotating the blog listings in much the same way as the sponsor logos are being rotated…
Anyway, regarding the statement that “analytics as core to CEP” - I confess that I just don’t see it as strongly as this myself. I think it might be more correct to say some/all of the following:
“You can plug analytic functions into CEP engines (required for some applications)”
“(Some) Analytics is an evolution of basic Real-time pattern matching native to many CEP engines”
In much the same way I would say that that Analytics is not “core” to a database, but is an important adjunct.
Regards,
Brian Connell
Brian’s comment brings up a good question for me. What is the definition of analytics? Is it purely a calculation, or something else? If one defines analytics as purely a mathematic calculation, then I would agree with the statement that some CEP use cases need analytics.
On the otherhand, if one defines analytics to include pattern analysis, one could argue all CEP require analytics. Simple mean, sum and average calculations is just one tiny piece of the larger analytics puzzle. Depending on who you ask, pattern analysis could mean tracing the causality, or it could be a statistical function that rates the detected pattern.
There seems to be a concensus that tracing the cause of a derived event is required, but that feels like a minimum requirement to me. A CEP engine that supports tracing cause, doesn’t necessarily support more advanced pattern matching techniques like machine learning, fuzzy logic or probabilistic logic. I feel there is still a long long way to go before CEP products reach the same level of maturity as other related domains.
Hi Brian,
Thanks for stopping by and commenting.
I disagree with your analogy about CEP and databases, and in particular, how you apply this analogy to the topic of analytics.
First of all, (structured) databases are a (structured) way for data storage. The actual storage and retrieval of the data in the database is by a structured query language. This structured query language is quite basic, focused on SELECTS, JOINS and SORTS, for example. It is primarily a set of technologies and standards for data storage and retrieval, not knowledge creation in the data-information-knowledge value-chain.
CEP, on the other hand, is a (much) higher level of abstraction; so to compare CEP and databases in the context of analytics is even more strange than comparing apples and oranges! (more like comparing apples and rocks).
The value proposition for CEP is creating (situational) knowledge from real-time data. Creating knowledge is a much different abstraction that storing (or retrieving) information. Hard drives store data, which makes them pretty low on the data-information-knowledge cognitive model. Filesystems provide a way to move data on and off the hard drive. Databases provide a level of abstraction above filesystems.
Without analytical thinking (processing) there is simply no CEP because you cannot abstract or create knowledge without analytical thinking. In other words, if there is no analytical thinking, there is no value creation.
We live in society where the value is in knowledge creation and knowledge management. Without knowledge creating, there is very little value. This helps explain why “CEP engines” have limited market and low sales.
The analysis of real-time complex events is the implementation of a model that represents (some) cognitive knowledge about a situation, opportunity or threat. The value is in the knowledge created. There is little value in a fast, slick engine that spends most of the time in the garage, in a manner of speaking.
Peter says, “I feel there is still a long long way to go before CEP products reach the same level of maturity as other related domains,” and he is absolutely right.
I simply go one step further in my commentary and say that many folks are already doing CEP, BUT the companies “doing CEP” are not the same companies that are callling themselves CEP companies.
I plan to post a table of companies that are doing “real-CEP” (all of them doing well); and none of them call themselves a “CEP company.” This will be in a future post.
The reason it is confusing to customers is that CEP is being marketed as something new, when in fact, only the three letter acronym is “new”, the concepts behind the TLA “CEP” are quite old and in some case very mature, much more mature than what we see in the CEP community.
Yours sincerely, Tim
Reading the above thread reminds me that a long time ago the combination of “data” (which often represents “events”) and “context” or (as it was called then) “domain knowledge” was the province of what were called “expert systems”.
I’m not claiming they are the same thing. “Expert system” seems to be a higher level of abstraction because it entails some specific domain while “context” could be “any domain”. But the two concepts “CEP engine” and “expert system” seem to be related if CEP can be aproximated with “SUM(events) + context”
Wikipedia describes analytics as a a science whereby “an entity arrives at an optimal or realistic decision based on existing data.” It goes on to state that “unless there is data involved in the process, it would not be considered analytics”.
I especially note the use of the term data.
In answer to the question on whether analytics is a core part of CEP, I believe that will depend on your definition of analytics. Many operations can take place on events without requiring analytics.
For example, there are many simple processing techniques that can be applied to events such as filtering, transformations, enriching, etc. More advanced processing techniques introduce spatial or temporal windows for performing running calculations or pattern detection. But at some point, the difference between an event represented in a window and persisted data in a data store becomes blurred and leads to confusion.
So a question: Is there an expectation that an event is transient? That it has a short life? That it is processed and discarded?
I use the terms “analytics” and “analytical techniques” to distinguish between situations where the entire data set is available (persistence of some sort) or not.
For example, you can perform analytics over a data set that has been persisted. It supports the series of regressions and refinements that make up powerful analytics. If your CEP engine persists all events within a window, I’d agree (hmmm..) if you stated that your CEP engine supports analytics.
Otherwise, I’d argue that your CEP engine applies analytic techniques, meaning that you support advanced applications of pattern matching with perhaps some enrichment, etc, but do not support the ad-hoc nature of most analytical techniques.
So, depending on your idea of the life of an event and on the nature of analytics, analytics may be possible.
You argue that the “value proposition for CEP is creating (situational) knowledge from real-time data.” Sure. But that’s only one value proposition. There are others. Not all create “knowledge” either, but it doesn’t detract from their usefulness or value.
You also state that “the analysis of real-time complex events is the implementation of a model that represents (some) cognitive knowledge about a situation, opportunity or threat.” Could this just as easily be translated as “threats, opportunities, and situations can be detected in real-time by recognizing patterns within various temporal and spatial windows”?
Perhaps to aid my understanding of your CEP model, one of the key questions is the “lifetime” of an event. Are you treating the event cloud as Yet Another Persisted Data Store (YAPDS) where events have (infinite) lifetimes?
Brian
If we go by Dr. Luckham’s definition. An event is persistent and immutable, so that implies inifinite lifetimes. I posed the same question on the CEP forum a while back and the responses I got was consistent with Dr. Luckham’s.
For the record, I don’t prescribe to the persistent immutable event definition. I think some events fits that model, but there are plenty that don’t fit. I like to joke that “not all events are created equal”.
Hi Peter,
I understand why events are considered persistent and immutable - for example if you treat them as records pertaining to “something that happened”.
But equally, I was posing the question/thought that if events have an infinite life, then we can use analytics to discover patterns. Looks a lot like standard data analytics to me…
Now if we introduce a concept of reacting once an event occurs - the real-time aspect - it changes depending on your point of view.
To some people, this is one-at-a-time processing whereby the “records” are records of some sort in a persisted data store, and the new event is a trigger that activates a query to check if a pattern is detected (implying that the query is processed against a full data set). Looks a lot like database triggers to me…
To other people, each new event is processed against a multitude of partially matched patterns to check if the new event completes the pattern. For me, this is the essence of CEP. Each pattern detected can result in yet another event that is perhaps a constituent element in yet another pattern, and so on.
In this case, the event has a possible short life through the CEP engine, and potentially longer (if not abstracted) if it is detected as part of a pattern. Either way, the CEP engine may decide to persist the event (and turn it over to the world of data, and all the existing mechanisms that exist) or to do nothing in which case the event “expires” (no record/data generated from this EPN).
Brian
Dear Brian and Peter,
Kindly refer to this post where we provided our original EP definitions to David and the (future) EPTS:
EPTS: Proposed Event Processing Definitions, September 20, 2006
http://www.thecepblog.com/2008/08/21/epts-proposed-event-processing-definitions-september-20-2006/
Kindly note that our Emerging Technologies engineering team at TIBCO (recall I formally worked for this group at TIBCO) provided early input defining events as “immutable”.
However, we did not (and do not) define events as both “immutable and persistant” because events can be persistant or non-persistant. Events generally should have a TTL (time-to-live) property.
We do agree events are “immutable” as a software design property, as we provided this concept, originally to David, Roy and the EPTS, as we recall the sequence of collaborative events in building the glossary.
Yours sincerely, Tim
My bias opinion is that for many cases “immutable” events are appropriate, but for other scenarios an event should be mutable. This brings up an important questions for the design and implementation of the pattern matcher. If a product goes with “immutable persistent event” definition, it simplifies the problem greatly and makes the reasoning process “read-only.”
On the other hand, if the pattern matcher supports mutable events, it has to handle modifications correctly, which is more challenging. Combine that with TTL property, building an efficient and robust pattern matcher for large datasets becomes a huge challenge.
I still feel a formal model for temporal logic and temporal distance is needed to make these things simpler, though not necessarily simple.
Hi Peter,
I will discuss why, from my perspective, and from a software engineering perspective, events are “immutable” in another post in the next few days.
Stay tuned, thanks!
(and look forward to your comments on that post!)
Yours sincerely, Tim