Cyberstrategics Complex Event Processing Blog

Paul Vincent of TIBCO wrote an outstanding post, The Value of (Production) Rules … Paul correctly notes:

In summary, of course,  event-processing rules, event-driven rules, and rules for business decisions can all overlap, depending on the application.

By coincidence, I was reading an excellent paper recently, Reactive Rules on the Web. In this Springer-Verlag paper, the distinguished ILOG and University of Munich authors (Bruno Berstel, Philippe Bonnard, Francois Bry, Michael Eckert and Paula-Lavinia Patranjan) say;

[Both] Event-Condition-Action (ECA) rules and production rules fall into the category of reactive rules, which are used for programming rule-based, reactive systems.

In other words, ECA rules and production rules are both a type of reactive rule, according to the literature.

Recently I read a blog post where the author hoped to distinguish between BRMS and CEP; citing a BRMS as a “production rule system” and CEP as a “reactive rule system”.   This distinction does not appear to work because production rules are also reactive rules, according to the literature.

If you are interested in event processing rules, I highly recommend Reactive Rules on the Web.

Filed under: Business Rules, CEP Terminology, Complex Event Processing, Event Processing | No Comments »

Seemingly inundated with blog posts about CEP and BRMS, it seems we should simply rename the current CEP space, BRMS.  All of the current self-described CEP products on the market today are rules-engines, this includes the continuous query stream processors and the RETE engines.  Furthermore, a quick review of Wikipedia says BRMS is, as follows (direct quote):

A BRMS or Business Rule Management System is a software system used to define, deploy, execute, monitor and maintain the variety and complexity of decision logic that is used by operational systems within an organization or enterprise. This logic, also referred to as business rules, includes policies, requirements, and conditional statements that are used to determine the tactical actions that take place in applications and systems.

The definition above does not say that processing events is not permitted.  Also, the description does not state that connectors and adapters to the event sources are not permitted in a BRMS.   In fact, Wiki goes on to state the following,

A BRMS includes, at minimum:

• A repository, allowing decision logic to be externalized from core application code
• Tools, allowing both technical developers and business experts to define and manage decision logic
• A runtime environment, allowing applications to invoke decision logic managed within the BRMS and execute it using a business rules engine

The top benefits of a BRMS include:

• Reduced or removed reliance on IT departments for changes in live systems
• Increased control over implemented decision logic for compliance and better business management
• The ability to express decision logic with increased precision, using a business vocabulary syntax and graphical rule representations (decision tables, trees, scorecards and flows)
• Improved efficiency of processes through increased decision automation

These are all the same arguments we repeated read from software vendors in the CEP space.

Why not rename the current CEP rules engines BRMS?

(The reason is quite simple, based on my experience.)

If software companies called all these new rules engines (all from players without a mature BRMS product) “BRMS” the marketeers would be required to compete head-to-head with the mature BRMS products and companies on the analysts magic charts and waves.

IBM has publicly called this space “business event processing”.  Since all the current generation, self-described CEP products on the market are types of rules engines, should we not ask the analysts to simply categorize all the current CEP products on the market BRMS?

Filed under: Business Rules, CEP News and Events, CEP Terminology, CEP Tutorials, Complex Event Processing, Event Processing, Event Stream Processing, IBM | 2 Comments »

The CEP Users Group I started on LinkedIn now has nearly 700 members.  This group was created to be a users group, not a vendor product marketing group or a group for headhunters to post their job searches.   For that reason, and to keep the group from future marketing spam, to be fair to all, and to avoid endless controversy over trivial discussions, this group now has a set of posting guidelines.

In early 2009 we will call to form a “users council” to advise the posting guidelines and expand the group moderators.   This council, or committee, will be composed of users and independent advisors across numerous vertical and horizontal sectors, including (for example) network management, security management, defense,  financial services, science, research, and transportation.

I sincerely apologize to any recruiter, seller or headhunter who may feel unhappy with these guidelines; however, we do not want the group on LinkedIn to become another marketing and positioning battleground.  We have enough of these battlegrounds, forums and blogs on the Internet, so we do not need to turn a professional network, LinkedIn, into another “who can out-position and out-blog link the other” network. This group was created for users, which means end users, independent consultants, analysts and solution providers to end users.  The group was not created nor intended to be a sales or marketing channel for sellers.

Guidelines for Posting in the CEP Users Group on LinkedIn

Please:

(1) No posts from headhunters or recruiters. These posts will be deleted and repeated violations will be grounds for removal from the group.

(2) No self-promoting posts from software vendors (to their software products or their links promoting their software products). This group is not for software marketing. These posts will be deleted and repeated violations will be grounds for removal from the group.

(3) The purpose of the LinkedIn group is to serve the User community, not the vendor community. For this reason, vendors that do not adhere to the guidelines may removed if deemed an abuser of user group privileges.

(4) Vendors or users who have questions or issues about these guidelines should contact the group manager(s) directly via email or by phone.

(5) There are ample other channels on the Internet (blogs, forums, wikis and more) for vendor marketing on CEP related topics. These guidelines are for the benefit of all; and also subject to change over time.

Thank you.

In addition, the group on LinkedIn is not a place to rehash the endless, and for the most part useless, “three letter acronym” (TLA) debates that arise and fall from time-to-time.   I agree with my colleagues that there may be folks who have never seen these debates, but then again, the debates are easy to find using Google search.

Frankly speaking, I have worked in senior IT strategy, implementation and architecture for over 20 years, and I have never worked in an organization where experienced folks debated about the use of terms, EDA, SOA, CEP, BPM, BI, ETC.  TLAs and alphabet soup discussions are great for analysts and marketeers looking to position their products and build their magical data sheets, but these timeless debates have almost nothing to do with solving real, operational business problems.  Folks solve operational problems with people, systems and code not TLAs.

This is precisely why, by the way, when a technology discussion is dominated by folks “selling” versus folks “buying” , the conversation generally degenerates to endless positioning chatter that serves little constructive purpose. These debates relating to CEP/EP have come and gone, repeatedly, over the past three years, and nothing has changed.  We need to hear from the buyers and users (Marc Adler of Citigroup comes to mind here), not the sellers.

If you are a CEP/EP vendor and your customer has a fantastic public use case for your CEP/EP product, please encourage your customer, the user, to share and post the information to other users.

Filed under: CEP News and Events, Complex Event Processing, Use Cases | 1 Comment »

In his Followup to “Do you need a Commercial CEP System?” Marc calls out Progress Apama and their great work in financial services.  I have always liked Apama’s product and what they are doing; however, we should be clear on the difference between marketing and technology.

Progress Apama was the company who, when the buzzword “CEP” burst on the scene, believed that the term “CEP” was not appropriate for these new classes of real-time event stream processing platforms.  Progress’ leadership, including Apama, worked very hard to reposition “Complex Event Processing” as “Event Stream Processing”.  I have always believed that Progress Apama, a company which prides itself in being run by engineers not marketeers, was correct, from a technical perspective, to call their platform “ESP”.  In fact, Apama still positions as ESP today, which makes me proud of them.

Then, the marketeers saw that, contrary to their marketing predictions, “CEP” was gaining traction, and they worked hard to reposition, yet again, and “ESP”  became “CEP” (again).   Some of the trouble with all this marketing spin and repositioning, is that it actually hurts good companies (like Apama, a company that prides itself on technical engineering excellence) because they become buzzword-driven, not technology driven.  It also is not good for the end users.

Marc goes further to say;

When I hear people at my firm talk about Apama, they talk about it as if it were its own class of applications, much in the way people refer to Ketchup now instead of Catsup. They don’t even know that Apama is related to CEP.

Precisely. Bravo!

That is because Apama (and other companies) make great products, but these forward-chaining, event stream processing engines are not processing complex events per se, they are processing events which are, by the most part, not complex, technically speaking.  Progress and the Apama leadership had it right when they fought tooth-and-nail to reposition this software as ESP engines.

The reason they “flip flopped” and repositioned, yet again, to the “CEP buzzword,” was based purely on marketing.  It has very little, if anything, to do with solving complex detection-oriented problems.

Now, regarding Marc’s question to me in his Followup to “Do you need a Commercial CEP System? …

However, there are many CEP applications in other industries, like gaming, transportation, and defense. All of them share the same characteristics. Monitor a bunch of real-time actions and warn when a certain condition is true. So, a generic surveillance block would be great. The surveillance block might come with a choice of adapters for the most popular sensors . (Tim Bass can tell me if I have been smoking crack here.)

Marc, you are right, in the general sense, but we cannot do all of these wonderful things you are talking about if all the “modules” are simply built upon forward chaining stream processors across a sliding time window!  Before we talk about the “intelligence” needed in these “expert modules” we need to move beyond the constrained thinking of the narrow world view of continuous query engines that can only do forward chaining.

We need CEP, not just ESP, and the products on the market are mostly just ESP engines.

TIBCO’s BE, for example, which is more complex than a simpler stream processor, is a forward chaining RETE implementation.   You cannot possibly tackle all the critical classes of complex problems that need to be addressed in defense, compliance, surveillance, detection and more if your modules can only forward chain and cannot backwards chain.

The analogy is quite simple.  If you have a set of alphabet building blocks and all the blocks are every other letter, you cannot spell very well.  The same is true with forward chaining rules engines.   Forward chaining rules engines are great, but CEP needs more than this, much more (neural nets, Bayes nets, backwards chaining rules, for example).

In other words, as I have often blogged, we are taking a set of technologies, with all their issues and constraints, and wrongly calling them CEP.  Why buy into the pure marketing hype based on the limitations of the software not the technical requirements of the customer’s real (and complex) problems?

Filed under: Advanced Event Processing, Business Events, CEP Terminology, CEP Tutorials, Complex Event, Complex Event Processing, Development and Evaluation, Event Stream Processing, Modelling and Simulation, Progress Apama, TIBCO | No Comments »

Sometimes I think Marc Adler is reading my mind, and I wonder how he does it.  I have been thinking for weeks about writing a detailed post about why Algorithmic Trading is not Complex Event Processing; but then Marc pens the thoughtful, Do You Really Need a Commercial CEP Engine?

In his post, Marc provides some interesting analysis and concludes that “if a commercial CEP vendor has a vertical application that solved our needs” the commercial CEP engine would be useful.  In other words, in Marc’s domain, what Marc seems to be politely saying is that if CEP vendors built robust algorithmic trading platforms they would be useful as algo trading platforms.

Frankly speaking, I experienced the exact same thing in Japan almost two years ago.  When we presented one vendors “engine” to the Japanese finanacial services client, their response was “We like [your company] and we respect you because of your messaging, but our clients want immediate, proven, robust algo trading out-of-the box.” Basically, all clients we talked to in Tokyo did not see very much merit in buying an engine that needed specialized programming, training and people from a company that is famous for infrastructure engines.  They wanted domain expertise not a “flexible platform” which is a fallacy promoted by software companies.

As Marc said, there are already myriad platforms for building algo trading applications and you can hire very clever C, C++, C# and other “down on the metal” system and applications programmers to do it.   The question I believe Marc could have also asked is next.  In addition to asking, Do You Really Need a Commercial CEP Engine?, which is a perfectly valid question, Marc might have also asked, Will Commercial CEP Engines Replace Algorithmic Trading Platforms?

Stream processing engines, marketed as “CEP” are actually not doing anything that many of us would remotely call “CEP” as was discussed when CEP was envisioned.  In fact, the class of detection-oriented, complex problems discussed in the original CEP literature is, for the most part, almost unsolvable by the stream engines marketed as CEP.    What is being sold is actually “event stream processing platforms”, which is very similar to what the The Securities Technology Analysis Center (STAC) calls this technology, as I recall.

STAC has the right idea.  STAC knows that these stream processing engines should not be called CEP products, because they are not really processing complex events.   They are right, 100%.  Just because your marketing department calls something “CEP” does not make it so.   Which leads me to my original premise, why do we find it necessary to rename algo trading platforms, “CEP platforms?”  We do not need to redefine the algo trading space as the new CEP space.  It simply is neither accurate nor useful, in my opinion.

I don’t think any of us have any problem with folks creating vertical algo trading platforms and intelligent order management systems.  This is a good thing.   However, it simply confuses the market to continally redefine CEP as something the software can actually do versus what CEP must do.  Processing streaming events and doing relatively simple pattern matches is not processing complex events.  As Marc shared with us, “our system does not do any sophisticated pattern matching yet. Right now, we are doing a lot of aggregations, groupings, rollups, and simple comparison testing in order to generate alerts and real-time dashboard information.”

This is precisely what clients have told me for two years.   “Aggregations, groupings, rollups, and simple comparison testing in order to generate alerts and real-time dashboard information” are generally only precursors to a true CEP application.   We have been doing similar things for decades with other tools.   These current generation platforms cannot, for the most part, detect nor process complex situations, and therefore, should not be called CEP platforms.

Filed under: CEP Terminology, CEP Tutorials, Complex Event, Complex Event Processing, Cyber-Trading Technologies, Event Processing, Event Stream Processing, Use Cases | No Comments »

Remember all the fun we had when we carved out race cars for our Pinewood Derby? For those of you are were not apart of this great American Cub Scout culture, the pinewood derby is a racing event for Cub Scouts in the Boy Scouts of America. Cub Scouts, with the help of their parents, friends and kits, carve and build their own cars from wood. Cub Scouts usually start from pinewood derby kits containing a block of pine, plastic wheels and metal axles. These events are so popular, the pinewood derby was selected as part of “America’s 100 Best” in 2006 as “a celebrated rite of spring” by Reader’s Digest magazine.

Well, it so happens I was doing some research into rule complexity and what defines “complex” and “complexity” in rule-based systems, when I ran across an article on rules and the pinewood derby. Here is a few quotes from Rule complexity (Rules, Part 2), which I thought you might enjoy.

When creating complicated rule sets, people tend to create many rules are subjective or are difficult to enforce. How do the race inspectors examining wheels ensure that they weren’t lathed? Are they weighing each wheel? Visually, lathed wheels can appear to be completely stock. Some races have prohibitions on cars from previous years or on pre-cut cars. The intent is to keep kids from using last year’s car. I agree with the intentions — building the car is part of the fun. But who can tell for certain which cars are pre-cut or a re-paint of their older brother’s car? Rules like this should be presented as guidelines that should be followed instead of rules that should be enforced.

We see so many software vendors running to sell us their lastest rule-based framework, and the network and blogosphere is buzzing with rule discussions.  So, keep this in mind, as our friend from the pinewood derby lamented:

The problem with trying to create a rule to cover every possible situation is that you can’t possibly think of every possible situation.  [...] — understand[s] that the rules can’t cover everything. The more complicated you make your rules, the more impossible they will be to enforce. You’ll introduce conflicts, ambiguities, and loopholes — [...].

I realize a few of my colleagues (most one that are are selling rule-based systems) wish I would stop blogging and just dissappear into cyberspace without a trace left behind.  They think I have lost my marbles because I cannot jump on the “hype up rules bandwagon!”    To simply disappear into the ether would be the easy road to go, because for anyone who actually has worked in complex operational problems (not just selling software or “talking the talk”), they understand very quickly that, as our friend at the pineword derby so nicely put it, you can’t possibly think of every possible situation, rules cannot cover everything.

Filed under: Business Rules, CEP Terminology, CEP Tutorials, Complex Event, Complex Event Processing, Event Processing, Situation Models | 2 Comments »

In response to a request via  LinkedIn to make my presentation for the AMCHAM Thailand ICT Cmte: Thailand’s Cyber Law Compliance Seminar available to non-AMCHAM members, Computer Crime Act B.E. 2550 (2007) & Ministry of ICT NotificationA Presentation to the AMCHAM ICT Committee & Internet Service Providers. You can also view this presentation on SlideShare.

Filed under: Cybersecurity, Education and Training, Thailand, Threats and Vulnerabilities | 2 Comments »

In GPS, GeoLogging and CEP I was really impressed with the movement toward multisensor data fusion and situation detection as commented by Alex and Marco.   Both Alex and Marco realize that basic track-and-trace tends to be a supporting role in many CEP applications; and that one core functionality that “graduates” an application to a “CEP application” is the degree of complexity in the object-object situation refinement.  (Note:  there are some classes of tracking that are quite complex, for example Kalman Filters).

It appears to me that both Alex and Marco tend to agree that simple rule processing, an important subclass of event processing, does not necessarily automatically “graduate” an application to the status of “complex event processing.”  However, as numerous readers have pointed out, processing rules can be quite complex.  So, this leads us to the obvious question:

What Defines Complexity in Rule Processing?

I have been thinking about this question, and do not have a good answer.   For example, I don’t think we can define “complex” as processing a simple rule or rule-based algorithm against high volumes of events (in a stream).  This is not complex, per se.  (Software vendors tried this early on and it went over like a lead balloon!)

So, what exactly would be the various parameters we could use to define, precisely, what is complexity in rule processing?

For example, in On rule complexity: A structural approach Dietz says:

This paper first examines existing accounts of rule complexity and presents a conceptual analysis of the term ‘rule’. It then proposes that ‘complex’ should not be equated with ‘difficult’, but used in a purely structural sense. Specifically a conditional formulation is proposed in which the number of concepts in the antecedent and the consequent, the number of subconditions, and the number of ‘if-then’ connections (subrules) within a given rule domain govern complexity.

In another example paper, Complexity Measures for Rule-Based Programs, O’Neal and Edwards said:

Software complexity measures are quantitative estimates of the amount of effort required by a programmer to comprehend a piece of code. Many measures have been designed for standard procedural languages, but little work has been done to apply software complexity concepts to nontraditional programming paradigms. This paper presents a collection of software complexity measures that were specifically designed to quantify the conceptual complexity of rule-based programs. These measures are divided into two classes: bulk measures, which estimate complexity by examining aspects of program size, and rule measures, which gauge complexity based on the ways in which program rules interact with data and other rules. A pilot study was conducted to assess the effectiveness of these measures. Several measures were found to correlate well with the study participants’ ratings of program difficulty and the time required by them to answer questions that required comprehension of program elements. The physical order of program rules was also shown to affect comprehension. The authors conclude that the development of software complexity measures for particular programming paradigms may lead to better tools for managing program development and predicting maintenance effort in nontraditional programming environments.

A quick survey of various authors on the topic of rules and complexity yields a number of interesting papers, but I did not come up with anything (yet) that would directly answer the question at hand,  What Defines Complexity in Rule Processing?

Please post your thoughts and references.

Filed under: Analytics, Complex Event, Complex Event Processing, Event Processing | 1 Comment »

Having been around the block many times, I enjoyed reading about SOA sinking into trough of disillusionment.

Let me tell you what is SOA, really….

SOA, in hardware terms, is the concept that if you take a perfectly good running computer, life, including your precious ROI, will be better if you:

• Share the motherboard as a service;
• Share the RAM as a service;
• Share the CPU as a service;
• Share the hard drive as a service;
• Share the video card as a service;
• Share the monitor as a service;
• Share the keyboard as a service;
• Share the mouse as a service.  … etc etc…

So, SOA, in software terms, means that you take a perfectly good running application, break it up into its components, and share the pieces as a service to gain ROI.

SOA is an amazingly effective architectural construct for taking working applications and making them inefficient, insecure and nearly impossible to maintain.  SOA also is great for fostering endless debates about “What is a Service?” and “Service Component Granularity.”

Sure, the core ideas of sharing software functionality as a service has merit; however, software vendors and analysts turned a “potentially useful concept” (PUC) into a huge sacred cash cow (HSCC) by making SOA into what it is today….

….  a Sacred Omnipotent Acronym !

Watch and learn, as we continue to see CEP going in the same direction, transitioning from “Complex Event Processing” to “Completely Elementary Processing”  … stay tuned!

Filed under: Complex Event Processing, EAI ESB & SOA, Humor | No Comments »

This is an excellent presentation on Internet trends by Mary Meeker, et al, Morgan Stanley.

Filed under: cyberstrategics | No Comments »