Muddy Waters

Posted on 07/16/08 11 Comments

In Waters Rankings 2008, Waters stirs the mud and confusion in the CEP/EP community by having their constituents vote on both an ESP solution and an CEP solution set, but giving both awards to vendors with stream processing (ESP) engines.   

The two CEP/ESP related Waters’ categories were, Best Streaming Data Management Solution and Best Complex Event Processing Solution.    Waters awards Best Streaming Data Management Solution to data/event stream processing company StreamBase; and then awards Best Complex Event Processing Solution to Oracle’s BEA product, which is built on top of another data/event stream processing engine.  Confused?   

Alexander Alves,  currently employed by Oracle, previously having worked for BEA Systems, in Best Complex Event Processing Solution, observes:

“Regardless, I find it intriguing that Waters not only does not state the differences between the categories, but also uses the term CEP several times in the SDMS category.

I guess the verdict is that there is still confusion amongst the experts regarding event and stream processing… And that both products must be very good.”

Of course, Alex must be politically correct, and rightly so, since he works for Oracle/BEA and Water’s gave them an award.  But on what tangible, objective basis for Best Complex Event Processing Solution?

According to our 2007 survey, CEP/EP Reference Customers 2005-2007, BEA was in last place based on public CEP/EP reference clients.  

Waters’ award  for Best Streaming Data Management Solution to StreamBase is more credible.   Congratulations StreamBase.   Most would agree that StreamBase is a streaming data management solution (SDMS), but so are Apama and Coral8 (and BEA etc etc).   

Waters simply muddies the water, unfortunately.  

Share and Enjoy:
  • Print
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks

9 Comments

  1. peter lin says:
    Wednesday, July 16, 2008 at 1:14pm

    Having looked at esper code, I question the ranking. Esper is an useful Open source project, but I wouldn’t rank it as top. There are significant design limitations of esper, which I feel make it inappropriate for many of the scenarios I know from first hand.

    I don’t know how IBM BPM got ranked as the top product either. The last time I looked IBM’s BPM I wasn’t impressed. Maybe they’re counting the “all ibm” shops that are stuck using IBM products. In anything, many IBM middleware products are painful to use and mind numbingly frustrating.

  2. Tim Bass says:
    Wednesday, July 16, 2008 at 1:27pm

    Hi Peter,

    Yes, these awards simply brings Waters credibility, objectivity, expertise and methodology into serious question.

    Yours faithfully, Tim

  3. Alex says:
    Wednesday, July 16, 2008 at 5:46pm

    Hi Peter
    I don’t know how far you researched around Esper, and I am curious on why you’d need to look at some product source code to get a taste on it – this would obviously defeat any other (non open source) product evaluation. Obviously Waters did not took the same methodology for its ranking.

    As far as I remember, we answered your questions on many of the public places (InfoQ, TheServerSide etc) where Esper was discussed and where you contributed questions, and there is largely room for further discussion (not FUD) on Esper dev and user forums if you want to get into more details regarding any limitation you might have encountered. As with any products – including IBM ones, there are lovers and haters, and there are support systems, open forums and identified point of contacts to discuss things in constructive ways if you want to.

    Also, bear in mind that Waters did not awarded Esper but BEA / now Oracle product. You might thus not balance Waters judgement vs your own personnal technical opinion about Esper as those are different products with a different ecosystem around.
    I’d also be curious to hear about your comments regarding other ESP/CEP products if you have by the way.
    Alex (contributing to Esper)

  4. Phil Albinus says:
    Wednesday, July 16, 2008 at 6:48pm

    Hi, just a word from Waters, the magazine in question.

    The Waters Rankings are basically our reader choice awards for subscribers who work for investment firms. We do not allow vendors or PR firms to vote and in fact, their votes are scrubbed out of the final tally. I assure you that the 600 voters for this year’s Waters Rankings chose the final winners.

    We have no influence over the outcome and if anyone feels that anything other than the deserving firm won, please make sure this doesn’t happen again next year by voting and encouraging your colleagues to vote. Not having used these products ourselves we rely on our readers to rank all products according to their own experience.

    Secondly, one person’s experience with a provider might differ wildly from the next. I love my Mac but I have colleagues that would never use anything but a PC.

    One of the challenges of Waters Rankings is the sheer number of categories and potential for sub-categories. We often have to merge some potential sub-categories into one category in order to make this a manageable survey for the voters. For example, we could easily have multiple categories for order management systems but we keep it to two: Best sell side and buy side OMS. At 22 categories, we feel that we are close to hitting the ceiling.

    If we were solely a CEP-focused publication, we would easily have multiple awards for the CEP-pies or whatever award would honor the CEP industry.

    This is obviously still a relatively new industry and terms such as CEP and ESP are often used interchangeably by the very knowledgeable people we speak with. There is a whole post on this site dedicated to solidifying the terminology of CEP/ESP. We welcome the feedback and hope to offer a clearer distinction in next year’s Waters Rankings.

    Phil Albinus
    Editor, Waters

  5. Tim Bass says:
    Wednesday, July 16, 2008 at 6:59pm

    Hi Phil,

    Thanks for stopping by and posting.

    Please point out one public reference client that is an investment firm that has announced an Oracle/BEA event processing solution.

    Can you post the URL to one public release where an investment firm proclaims they are using the Oracle/BEA event processing solution?

    Perhaps the Waters voting methodology is flawed?

    Your faithfully, Tim

  6. Phil Albinus says:
    Wednesday, July 16, 2008 at 7:14pm

    I fail to see how our methodolgy is flawed. The readers help create the list of nominations and then choose the winners. We do not ask for URLs of press releases and firms are loathe to say which solutions they use, as you well know.

  7. Tim Bass says:
    Wednesday, July 16, 2008 at 7:30pm

    Hi Phil,

    Just as you say “firms are loathe to say which solutions they use, as you know well” we can say, in reply, that “software vendors are keen to encourage users to vote for their products in award votes, which you know well.”

    Depending strictly on voting is a flawed methodology in a world where vendors can influence the votes, and larger vendors have greater reach and influence, as you know well.

    Regarding reference clients. Yes, users are loathe to agree to act as reference clients, that is why when vendors have public reference clients it is very noteworthy.

    So, personally speaking, I would trust a number of major customers standing up and saying “we use a product to solve this use case” versus a group of users who may or may not even use the product, voting because they are being encouraged to vote.

    Also, you should look at other factors, such as time in the marketplace, etc. if you want a more realistic and accurate methodology. Voting can easily be influenced by people who don’t even use the product in practice.

    Yours faithfully, Tim

  8. Peter Lin says:
    Wednesday, July 16, 2008 at 10:55pm

    Hi Alex,

    thanks for posting a response. I looked at Espers code to see how it compiles queries to pattern and how it handles complex queries. I also took a closer look at the cube functionality in Esper to try to understand how it differs from existing technology. You have access to my blog, so I would suggest reading my recent entries regarding continuous versus on-demand aggregation. Since there’s a size limit on comments, I can’t respond fully in a single comment. I would highly suggest reading up on sparse cube issue and precalculation issue with cubes. Based on my understanding of how Esper uses continuous queries with cubes, Esper takes a pre-calculated approach. Is that an accurate interpretation of how Esper works? If it is, my blog entry provides several long posts that go into detail.

    thanks

    peter

  9. Peter Lin says:
    Wednesday, July 16, 2008 at 11:00pm

    Here’s a few links to recent entries on different types of aggregation.

    http://woolfel.blogspot.com/2008/07/simple-time-window-versus-temporal.html
    http://woolfel.blogspot.com/2008/07/pre-calculated-and-on-demand.html
    http://woolfel.blogspot.com/2008/07/to-pre-calculate-or-not.html
    http://woolfel.blogspot.com/2008/07/more-thoughts-on-aggregations.html

    Here is a link to my paper on temporal logic, since one of the entries mentions temporal logic.

    http://jamocha.svn.sourceforge.net/viewvc/jamocha/morendo/doc/Temporal_Logic_Extension_for_RETE_Algorithm.doc?view=log

    I look forward to hearing your response on how Espers address the sparse cube and pre-calculation issue.

    peter

2 Trackbacks

  1. [...] glossary should also useful in defining the CEP sector for outsiders given the widespread use of apparently interchangable phrases such as Event Stream Processing, Event Processing, Stream Processing and Complex Event [...]

  2. [...] Muddy Waters [...]