The Complex Event Processing Blog

CEP and the 1984 Apple Macintosh Commercial

As someone who has been in technology all of his career, I have seen a lot of “not invented here” barriers to progress.  I think we all have.  All of us have “NIH” stories and the terms “not invented here” and “NIH” are common place.  This phenonema is so common, there is even a Wikipedia entry, where the authors say:

Not Invented Here (NIH) is a term used to describe persistent sociological, corporate or institutional culture that avoids using or buying already existing products, research or knowledge because of its different origins. It is normally used in a pejorative sense.

As a sociological phenomenon, “Not Invented Here” syndrome is manifested as an unwillingness to adopt an idea or product because it originates from another culture, a form of nationalism.

Event processing has been around as long as I can remember; and this means at least 22 years of my career as a systems engineer and consultant (not to mention a one year and a half “global experiment” with my favorite software company).   As far back as 1987 we were collecting network events and trying to make sense out of complex situations.  We wrote rules and more rules against events.  In the beginning, most of the rules were against SMNP events in networks.  We later expanded this to security events.

By the time 1993 rolled around, 16 years ago, we were working on new ways to aggregate, filter and correlate network and security events.   Rule based systems were always limited in utility as we discovered in the day-to-day operations centers of USAF network command centers during real-time hackers attacks.   We searched for better ways, welcoming any and all technologies.

Now we find ourselves, in the year 2009, and we still face the same problems.   I cannot take any so-called “complex event processing platform”  (CEP or EPP)  today and do anything that we could not do with PERL scripts 15 years ago.  We had systems programming, sockets, adapters, filters, aggregators and a lot of rules.   Our rule sets were so large that they were ill suited for a graphical layout, it was easier to use a UNIX grep filter to find the rule and modify it.   The best programmers user VI and EMACS, not MS Word.

Working on public, unclassified military networks, I searched for better ways and introduced the concepts of multisensor data fusion to computer security in a series of papers in the late 1990s.   I never claimed ownership of these ideas.  I simply combined and synthesized ideas across multiple disciplines.   This is how systems engineers tend to think anyway, we make no claims other than being humble systems guys.

In the beginning of my work on computer security and fusion there was resistance, mostly from a small group at Stanford (ironically) working on a classification scheme for computer security.  This group was  unhappy with my fusion approach and discussion of advanced analytics , inference and so forth and so on.  (Or perhaps simply unhappy that I was slowly getting more attention than they were getting.) Now, of course, sensor fusion-based network security is one of the most researched and developed areas and the number of papers and applications are countless.   Computer security has not been the same since.

Then, while at TIBCO, was introduced to the phrase “complex event processing” and I thought, “Wow, this has potential.“  I jumped around the globe  in airplanes and had a nice “principal global architect” gig with TIBCO, a great company.  TIBCO supported me and they never forced me to parrot a party line.   True innovative thinkers.  I loved working there (but my future plans were to move to Asia).  In fact,  TIBCO engineers often discussed advanced analytics, fusion and how this effects CEP.

I briefed the wider “CEP community” in March 2006 and found these folks in the middle of debates about terms like ESP and CEP.  The entire world of event processing (in that group) seemed to be focused on database joins and streaming queries using a rule-based approach across time windows.  I mentioned how this is perfectly okay for a subset of problems, but it is far from what my customers needed.

Here we are, 3 years later, and the same vendors are trying to convince folks like me that they invented the technology or idea to solve complex events.     I have been told recently there is but one “CEP authority.”   All these new faces and companies on the scene, none of which have a technology  which will solve the classes of problems we were working on 15-20 years ago, or even today.

Perhaps, I lament, there have been other technical communities where NIH was bigger and more intensive than the CEP community, but I cannot recall one, off-hand.    Of course, when you are experiencing some pain, the pain tends to feel like the “worst you can remember” so I am hesitatant to conclude that the small CEP community is the most intense NIH community I have even seen.   It is really quite small.  It is so small, in fact, that we rarely hear from customers or true end users.  Most of the CEP chatter is either from researchers, vendors or vendor cheerleaders.

What is troubling to me lately, however, is how little anyone in this co-called CEP community discusses any prior art or even complimentary modern art.  Almost all the discussions are centered on finding ways to promote the capabilties of a very narrow technical implementation of an approach similar to what we were using 20 years ago.   When we try to change to subject to advance analytics, methods that actually might detect “complex event,” we are told to “go away”.  Folks, or at least a small group of them, demand we “create our own term” because they “own” CEP already.  Nevermind they can’t detect a complex event, but they can sure route a market order!  Yea!   Go Market Order Go!

I don’t get it.   I have never been into NIH and have always rejected it.   This new CEP community, however, seems intensively NIH.    They have placed the word “complex” in front of the words “event processing,” and so therefore, they invented “processing complex events.”   Terms in science and engineering like “complex systems”  and “detection theory” have been completely discounted, as if because they were invented elsewhere, they are irrelevant.   The prior art, engineering and science outside of a narrow “newly invented view” is totally glossed over with “Oh Yeah, There was Event Processing before in many forms and disciplines….” and then, like a rubber band, everything snaps back to “comfort zone” NIH again.

We are basically told not to question “authority” – we must “respect authority”.  It makes me feel like I am llving in  Orwell’s Animal Farm where we must tow the party line right or wrong, or leave the farm.   Maybe the better analogy is the movie “One Flew Over the Cuckoo’s Nest” where, as I recall (perhaps incorrectly) you get a lobodomy so you “fit in” to the culture of crazy.

Anyway, I digress

I need to go across the street to the hospital and ask the doctors to give me a lobodomy and remove all the prior knowledge in my brain, all prior experiences, and simply worship “authority”.    I want to be like the drones in the that famous Apple SuperBowl 1984 commercial.

CEP.  CEP.  CEP.  CEP.

Believe.  Believe.  Believe.