CEP is Not a Just a Technology and Not Just a Tool

Posted on 11/10/08 7 Comments

I read the debate (here, here and here) on how Complex Event Processing (CEP) fits into the wider software architectural themes of Service Oriented Architectures (SOA) and Event Driven Architectures (EDA).  More comments and blog posts followed (including this one, and this one).   Frankly speaking, I was surprised to see so much misunderstanding on fundamental concepts.  For example, Giles Nelson blogged:

1. CEP is a technology. SOA and EDA are not technologies. SOA and EDA are philosophies for the design and build of modern distributed computing architectures.

The statement “CEP is a technology” is a bit misleading, demonstrating how little folks understand CEP and how quick they are to reduce CEP to a “technology” that fits their software solution, versus a design philosophy for solving complex problems.

I generally agreed with the overall concept of what Jack van Hoof tried to say; however, Jack also implies that CEP is just some small piece of the puzzle that is a part of an overall EDA approach.  This is both right and wrong.

First of all, you can perform CEP without an EDA, so folks in the debate are comparing apples and oranges.   EDA is a design pattern, or philosophy, for processing events with extreme loose coupling.  CEP is a design pattern, or philosophy, for creating complex situational knowledge from a cloud of events.  CEP can be, and has been, performed with or without an EDA or an SOA. See, for example, this presentation, Event Driven Architecture (EDA), November 2, 2006.

Mark Palmer add his inconsistently wrong on view on CEP into the discussion:

“We didn’t use CEP because it didn’t exist.  Until recently.”

This is absolutely false.  Every major voice in CEP/EP has stated that CEP/EP has been around for a  long time.  What did not exist, until recently, was the three letter acronym “CEP” and the handful of niche player vendors positioning their products as “CEP”.  In fact, processing complex events has been around as long as I can remember (more than 20 years on work in IT networks), especially in the network and security management space.

Of all the voices in the debate, Opher Etizon correctly points out that CEP and EDA are orthogonal.  Opher has been consistant and correct in this point.  You do not need an EDA or SOA to do CEP and that is a fact.  Nor do you need CEP to do EDA or SOA, and that is a fact.

David Luckham then asks,

Hmmm. Someone asked me a while back if “CEP was going to get ‘confused’ like SOA is now”.
Might be worth a community discussion on how to try avioding that happening! Or can we?

In reply, frankly speaking, CEP has been “confused” since the term was minted, as I pointed out in my first keynote talk on event processing, see slide 4 of my presentation, Processing Patterns for PredictiveBusiness, titled A Vocabulary of Confusion.

Share and Enjoy:
  • Print
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks

7 Comments

  1. Mark Palmer says:
    Tuesday, November 11, 2008 at 12:50am

    Hi Tim – Yes, complex events have been around for a long time, just like data was around long before the RDBMS existed. But generalized CEP platforms are new. If you think that statement is wrong than you should simply cite a generalized commercial platform for CEP that existed before 1998. Fill the name of that platform in here: _________

    To clarify, my posting (http://streambase.typepad.com/streambase_stream_process/2008/10/eda-and-cep-wheres-the-beef.html) said we did event processing in 1988 on our Banker’s Trust trading system. Sure, we detected “complex events” in our software. Of course we did! But it was hard to do, and we did it with a lot of C++ code.

    We didn’t use a general CEP product like StreamBase because no such platform existed. I wish it had existed. Unfortunately for us, general CEP platforms were commercially available for the first time in the late 90′s. So they are new.

  2. Tim Bass says:
    Tuesday, November 11, 2008 at 5:16am

    Hi Mark,

    Well, you continue to shift your position, and write clairifications. You did not say that “CEP platforms are new” in your post. You said “CEP was new”. So, please be more precise with what you are saying, thanks!

    Now, to your point that “generalized CEP platforms” are new; that is also not true.

    First of all, your product is a rules engine designed for continuous streaming queries over a time window.

    Folks can do just about the same thing with PERL, a language that has been around for a long time. PERL is a generalized tool for extracting patterns from streams of data (and events).

    Hence, you are jincorrect in your revised statement that generalized platforms that process streaming events or data are new. We were using PERL in 1998 to define the USAF against cyberattacks, where we detected patterns of network abuse in real time.

    The problem was not the language, or the generalized platform. The problem was that rule-based systems have scalability problems. Your engine would not have helped us ten years ago, we already had a generalized tool for extracting patterns from streaming event/data, that tool was called PERL.

    You are a good marketeer and a talented writer, but you are continually posting misinformation about the history of processing complex events.

    Yours sincerely, Tim

  3. Tim Bass says:
    Tuesday, November 11, 2008 at 5:39am

    Also, to follow up to my reply to Mark Palmer’s comments that “generalized event processing plaforms are new”. Kindly turn your attention to the Wikipedia entry on event-driven programming:

    http://en.wikipedia.org/wiki/Event-driven_programming

    There are many generalized event procressing frameworks, many more than 20 years old.

    Many of these frameworks have much more event processing capability (features) that StreamBase’s product.

    Cheers.

  4. calmglow says:
    Sunday, November 16, 2008 at 2:15am

    Hi Tim.
    Thank you for your good article.
    But I think you’re too strict and it’s argument for the sake of argument. I think the CEP has old and new things. you’re maybe focusing the old area of the CEP.

  5. Tim Bass says:
    Sunday, November 16, 2008 at 3:11am

    Hi calmglow,

    You mean the “old area of CEP that was never addressed”? Believe me, there is no argument for the sake of argument here. I have no software to sell you. Be careful when buying into hype and marketing associated with buzzwords.

    Yours sincerely, Tim

  6. Roland de Boo says:
    Tuesday, January 6, 2009 at 12:09pm

    Hi Tim,
    You state that “You do not need an EDA or SOA to do CEP and that is a fact. ”

    However in the presentation you linked, “Processing Patterns for PredictiveBusiness”, slide #3 states:

    “CEP requires a Number of Technologies:
    *Distributed Computing, Publish/Subscribe and SOA”

    Is this something you have changed your mind about, or an error in the slides, or something else?

  7. Tim Bass says:
    Tuesday, January 6, 2009 at 2:24pm

    Hi Roland,

    Thanks for visiting and for your comments.

    Yes, when I was working as a principal global architect as TIBCO, many of my slides were influenced by TIBCO technology focus areas and TIBCO corporate objectives.

    I still view the “macro” architecture of CEP as a distributed architecture, and in the general case it is a distributed architecture, because we assumed that events were distributed in an overall enterprise architecture; and processing events, in the general case, is a distributed architecture.

    Today, I would rephrase my slides, if I had them to do over again, to be more general, to say:

    “CEP Benefits from a Number of Technologies:

    *Distributed Computing, Publish/Subscribe and SOA”

    It is not a hard requirement for CEP, or event processing in general, to be pub/sub or based on an SOA. So, my thinking has evolved since leaving TIBCO, a great company that specializes in pub/sub architecture, EDA, SOA and similar state-of-the-art software architectures.

    Naturally, my original slides were influenced by my senior position at TIBCO, the TIBCO philosophy and the TIBCO technology stack. EDA (and pub/sub) and SOA are concepts which I now consider to be interesting and complimentary to CEP/EP, but, the the most general case, not a hard requirement.

    Distributed computing is generally required to enterprise classes of event processing problems. Publish/Subscribe (or EDA) is normally a part of an enterprise solution, as is an SOA. However, these capabilities are not required for 100% of all CEP/EP applications, generally speaking.

    I hope I answered your query in a satisfactory manner.

    Yours sincerely, Tim