Orwellian Event Processing

Posted on 02/28/10 16 Comments

Recently we completed the installation and training of an open source Bayesian classifier to replace a rule-based approach to manage forum spam.  In a nutshell, we found the rule-based approach was highly prone to both false positives and false negatives; however, a statistical approach using a Bayesian approach has turned out to be far superior.  We are applying this same approach to real-time threat analysis and other classification problems.

The engineering question is not “should we completely get rid of rules and replace rule-based approaches with more sophisticated analytics?”.   Rules are useful and work good for many simple processing problems.  However, rules alone are highly inefficient for most classes of (not simple) problems.   I have pointed this out a number of times over the years and I think most people “get it”; so I was a bit surprised when I read this post last year by Paul Vincent,  CEP versus ESP – an essay (or maybe a rant).  In that post, Paul blogged:

“The wider “complex event processing” term additionally covers other mechanisms like ECA rules, production rules, and so forth …”

I think the industry would be a lot better off (grow faster, solve more problems, be more profitable) if folks selling hammers would cease to define the world based on what can be tooled with hammers; and folks who sell screwdrivers would stop defining the world based on what screwdrivers can do well.  To Paul’s credit, he does conclude, correctly in my view:

…in most enterprises there are usually multiple use cases for multiple types of CEP that are best handled by multiple paradigms (such as specialist ESP, event-driven business processes, rule-driven event processing, event-based business rules, event-driven analytics, etc). One should no more expect a large company to rely on a single CEP paradigm as it would on a single computer hardware technology.

This brings me back our  various classification projects, one is documented in A New Bayesian Spam Classifier Using B8.  In that project, we used some simple rules to pre-process text, but the more sophisticated and complex processing is performed by a statistical classifier.

Let me simply conclude by voicing my continued frustration at anyone who believes that rule-based (or query-based) approaches are a metaphor for complex event processing.  They are not.  Rule and query-based approaches are more closely aligned to simple event processing.   Writing IF-THEN-ELSE logic is quite simple.   Adding a new condition to an IF-THEN-ELSE statement is also simple.    The only thing complex about this approach is managing a large set of rules, because the more complex the problem the more unscaleable and difficult to manage any rule-based approach becomes.

Conversely, the opposite can be said to be true with systems that are specialized in complex event processing.  Complex data sets become training sets for more advanced statistical methods.  In fact, “the more the merrier” is a good way to describe it.

For example, if you have a rule or query processing system and a new condition appears, your system will experience either a false positive or false negative.   Then, you must go write a set of rules to manage that new condition.  That new set of rules might adversely effect your existing rule base (we have seen this in practice) and cause an unexpected false positive (or negative) later on down the road.  Rules are simply not efficient in complex data processing solutions.

However, when you use advanced analytics, like a well designed Bayesian classifier,  and a new condition appears, it is not necessary to write any more logic. No coding.  No new configuration.  No new rules.  You simply send the new condition(s) to the classifier and the system “learns” from the experience.

We would all be better off if the folks in the CEP space (including my friends, and ex-close friends, at TIBCO, StreamBase, Progress, etc) would stop using CEP as a metaphor for rule and query-based event data processing.  In fact, the opposite is more likely true. True “complex event processing” is the processing task that rules do not perform efficiently – the software that Paul Vincent (and others) marginalizes as “and so forth” because they  work for a company (or companies) that are selling “a hammer” (rule-based software) and so therefore everything out there must be defined as “a nail” (a problem that can be solved with rules).

This is one key reason that CEP, the term and the technologies, continues to flounder and sink.  Customers and end users need more sophisticated methods, but the vendors keep trying to tell us that “simple” is “complex” and “complex” is “simple”.    Only software vendors, analysts and advertisers sing the praises of CEP because most complex problems cannot be efficiently solved with rule or query-based approaches (alone). The new users, the people with the complex problems, are not “buying the hype”.

Perhaps we should rename the CEP space “Orwellian Event Processing” ?


Orwellian:

‘… describes the situation, idea, or societal condition that George Orwell identified as being destructive to the welfare of a free [professional] society. It connotes an attitude and a policy of control by propaganda, surveillance, misinformation, denial of truth, and manipulation of the past.”

Share and Enjoy:
  • Print
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks

16 Comments

  1. Charles Young says:
    Sunday, February 28, 2010 at 5:07pm

    Oh please Tim! After several months of quiet on this front, do we really need to start going around these loops again? I guess we do.

    a) You continue to equate the simple syntactical forms of rule expression with simplicity in the function of those rules and the problems they can be applied to. Confusing form with function is a common mistake. In reality, the search for elegant design is often a necessary precursor to building practical tools that can handle complexity. Do not misunderstand me. I do not claim that rule processing automatically yields elegant approaches to all the problem domains to which it is applied. Sometimes it does, sometimes it doesn’t. In terms of form, the search for greater elegance and simplicity continues. The main driver for this search is to allow us to handle greater degrees of complexity in a practical (and cost-effective) manner. Certain types of rule processing can handle certain types of complexity in a very elegant fashion.

    b) You continue to contrast a statistical approach to reasoning under uncertainty (Bayesian analysis) with rule processing as if these two things offer head-to-head alternatives to achieving the same ends. I can’t imagine what your thinking is based on. Obviously, if you need to reason under uncertainty, you ought to select the best tools and techniques for this task. Rule processing is not fundamentally conceived as an approach (statistical or otherwise) to reasoning under uncertainty. Bayesian analytics is. For decades, rule processing has been used in conjunction with Bayesian analytics, Dempster-Shafer fuzzy logic and other approaches to handle uncertainty. Read virtually any academic book on rule processing to see how these approaches are combined. Here’s one from my bookshelf – ‘Managing Uncertainty in Expert Systems’ by Jerzy Grzymala-Busse published in 1991 by KAP. It follows a well-worn path. Back in the early 1980s, for example, most rule-based expert systems (even those built for microcomputers) already supported approaches such as certainty factors (a poor approach – see http://www.aaai.org/Papers/AAAI/1987/AAAI87-022.pdf), Bayesian analytics or Dempster-Shafer.

    c) Rule processing is not used as a ‘metaphor’ for CEP because CEP is, in fact, a form of rule processing and is therefore fundamentally the same concept. David Luckam is quite clear about this. See, for example, the opening paragraphs of chapter 10 of Power of Events or, for that matter, the whole of chapter 9 entitled ‘CEP Rules and Agents’. As for other forms of rule processing, it makes lots of sense to exploit approaches such as Bayesian statistics within the context of CEP in order to reason under uncertainty.

    We would all be better off if people stopped throwing dingbats at each other and instead concentrated on applying the lessons to be learned from many different disciplines and working out elegant ways of combining those multiple disciplines in order to tackle ever-more complex problem domains.

  2. Tim Bass says:
    Monday, March 1, 2010 at 6:22am

    Hi Charles,

    Thanks for your passionate comments. Yes, we are going to start this again… :D

    Actually, I think it is accurate to say that of most all the vocal people who discuss and blog about CEP, I may the one of the few who actually have experience building these systems in practice that have to detect anything. (( There is a Popular Science article written about my work in cyberattack countermeasures dating back to years before the phrase “CEP” was coined, LOL. — Did you read it? Did you read about our team “writing rules” 13 years ago?? ))

    To contrast your reply, here are excerpts from a private email just in on this post, perhaps the writer was afraid of being attacked by those have little clue about what they are talking about, or those who write books that have rarely been in a data center or ever actually worked on a large scale project to detect anything.

    ———–Excerpt from note just received: ——

    Like you Tim, I have issues the term CEP. In fact, I’ll go one step further and say “CEP is a stupid term”. What most of the CEP products handle today is just stream processing. The term event processing makes absolutely no sense to me. Applying the term “event” to either stream processing or complex scenario is just pointless. I say abandon the term CEP completely, for the simple reason that it’s garbage.

    I agree with Charles Young on many levels. Bayesian filters, kalman filters, fuzzy logic, production rules and statistical analysis has a long history of working together. The problem is, many people in the CEP world have zero understanding or experience using them. That in itself is expected. What I dislike is when CEP proponents make statements to try to silence dissenting voices, out of fear. There’s a wealth of prior art waiting for those eager to learn. Too bad many only care about making a buck and ignore prior art in favor of profit motive. This is true of proprietary and open source software.

    ….. (snip, protecting the identity of the sender) ….

    All around, there’s lots of confusion. …. Unfortunately, there are people in the field calling themselves “experts” in rule processing, production rules, business rules, ECA rules and event processing with only 5 years of experience.

    I’m thankful you’ve started blogging again. For a while there I was afraid you’d stopped blogging because of the attacks from Opher and those who side with him.

    —– (end of cut and paste of quote) ——–

    So Charles, I think you should stop quoting people who, to my knowledge, have never worked in an operational data center or written a single line of detection code (maybe they walked through one on a tour, LOL). Just because someone coined a phrase “CEP” does not make them an “expert” in actually processing and detecting complex events. In fact, most of the “book authors” and others in this space have zero operational experience in detecting anything, and certainly nothing “complex”. You are quoting the non-warriors who writes about war, Charles. Don’t you see the irony?

    The more accurate metaphor are “experts” writing books on how to police and never, ever putting on the uniform and solving a crime or protecting a citizen; or “experts” writing books on the theory of war and never fighting in one (win or lose). This is what has been going in the CEP space. Just like someone posted to me privately above:

    “What [the person who replied] dislike[s] is when CEP proponents make statements to try to silence dissenting voices, out of fear. There’s a wealth of prior art waiting for those eager to learn. Too bad many only care about making a buck and ignore prior art in favor of profit motive.”

    This is a fact Charles. The “leading voices” in the CEP space, almost all (if not all) have zero operational expertise detecting anything “complex” and all most care about is “making a buck”. Personally, I find it quite a joke that there are at least six leading “CEP people” in the space who have never (ever) worked in a modern, operational environment where they had to detect anything critical or complex in a critical IT system!!

    What gives them the “right” to attack the actual users (like me and others) who have spent most of their lives in actual operational projects?

    The fact of the matter is that we can detect complex events with free and open source code better than any commercial CEP software on the market… and I am a user, not a seller and don’t need to a make a single buck on CEP software. This is a fact. I worked for a software vendor for a little over a year to “see what is was like” and “for the experience”…. do you really want to know what I learned about software marketing, software analysts and their “experts”?

    Why are you so “pro CEP”… what complex event (or situation) have you detected lately Charles?

  3. Charles Young says:
    Monday, March 1, 2010 at 8:13pm

    Tim, you are not the victim here! You have a long history of making provocative comments. Don’t complain if they get a reaction!!

    If you review the comments I have posted previously on your blog site, you will see that I have always responded to what I consider to be a mistaken characterisation of rule processing in general, and not CEP in particular. I do, however, believe that rule processing remains an important component of CEP.

    In my previous comment, I suggested that you have a) mistaken form with function regarding rule expression, b) confused a statistical approach for reasoning under uncertainty with a (semi-) declarative approach to reasoning via predicate logic and c) failed to acknowledge that CEP, as commonly defined, is a form of rule processing. I quite understand if you do not wish to engage me in these issues, but understand that I equally won’t choose to meet you purely on your terms. Give me a reasoned debate, and we will have something to discuss.

    Like you, I am a user and not a vendor. I can assure you I am not reacting out of fear or direct commercial interest. I do suggest that you are mistaken about certain aspects of rule processing and that some of your arguments are therefore not well-founded.

  4. Tim Bass says:
    Tuesday, March 2, 2010 at 9:27am

    Hi Charles,

    My kind advice to you is that if you want to have a professional discussion, you should not start your replies with (...snip…) “Oh! Please Tim!” and “You are not the victim here...” statements. Thanks.

    Frankly, it seems you are losing your objectivity.

    I certainly may not have as much experience in Microsoft based systems as you do (I’m a Linux/Unix person), as you are constantly singing MS and BizTalk praises in your blog and I may not have any Microsoft BizTalk expertise as you do; but I certainly have years and years more operational experience in network and security management than you and much more expertise in UNIX and Linux. None of us know everything about all topics, and you are no exception, and neither am I. So, if you want to find a weakness in my logic and want to use that as a counter attack point, please do not feel unhappy if I also use the same type of weaknesses in your background to counter.

    Your posts and replies seem to come from a theoretical perspective deeply entrenched in a MS worldview. You rarely reply to any of my direct technical questions or challenges; and instead you have recently started more of an emotional approach (“Oh, Tim Please!) to this discussion v. a logical one.

    However, I will attempt to put this discussion back on track and I hope (and ask) you will reply directly and avoid the ad hominims, that Mr. EPTS et at use when they do not have a supporting technical reply. I have seen this all too often, when a person does not have the background or basis to counter in a debate, they resort to ad hominims. Many have commented on this before in the past CEP discussions and have shown considerable support to me as I have attempted to stick to operational facts and issues. So, I will start again here:

    In 1998 the USAF was attacked in a large scale distributed cyber attack that took advantage of a hole in how AF configured their main mail relay servers (this is old news today). I led the technical countermeasures at that time (at Langley AFB) and we initially used a rule-based approach I developed in Perl, to detect and respond to massive email bomb attacks. Many in the DoD claim I wrote the first cyber defense code against a bone fide cyber attack. This is all documented publicly.

    When the attackers discovered their email bombs were not getting past our rule filters, they began to make small changes in various parts of the data. Many of these changes resulted in a missed detection and another rule had to be written. At the height of the battle, we had around 5 people writing rules full time. Yes, we won the battle, and yes my work was featured in IEEE, .login, CNN, Federal Computer Week, Popular Science, briefed to the top US cybersecurity commission at that time, and countless other places; but I knew, first hand, the rule-based approach was flawed. It worked, but it was very ineffective and resource intensive.

    That is when I began looking into the prior art of multi sensor data fusion and wrote an ACM paper on using this approach for cyber security. Now, this type of fusion approach is the leading approach to network security. For a while, security companies were calling their products “fusion this” and “fusion that”…. These are facts, and based on experience, in real operations, in real time, nearly 12 years ago. I have not been asleep since that time, LOL.

    When I read your background, it is impressive. You seem to have a lot of experience (or interest) in Microsoft and BizTalk. You are passionate about BREs and some of your closest associates are in the rule space. I have not seen any operational experience in the “needle in the haystack” type of detection problems that were the genesis for DARPAs work on CEP. However, I did not see your resume on LinkedIn, so maybe I am missing something. My comments are from my reading of your blog posts and replies here.

    When I read your blog, or send your RSS to other sites I manage, I see mostly posts on Microsoft, rules, BRE and BizTalk. Those are fine areas, but they don’t make you an expert on detection, open source, Linux/Unix based processing, PHP, Perl, network management, or security management. In fact, I have never read anything from you where you discuss the basics of detection theory, how to solve “needle in the haystack” types of problems, issues with false positives, false negatives, and other basic detection terminology.

    So, from my view and experiences and interests, which is obviously different that yours. I view your posts and replies about rules as being mostly theoretical with little operational foundation in detecting anything DARPA (or the DoD or the USAF) would consider “complex” , nor are your replies applicable for any of the classes of problems that DARPA funded Stanford University to address on this topic. You certainly do not address these issues technically in your blog or here. You have shown a more recent interest (on your blog) in statistical methods, but I don’t think you have yet applied these terms to a large scale, real world, situation.

    Regarding the current CEP IDEs, I have download and tried to use various CEP IDEs and found that none of them worked better, faster or cheaper, than using Perl or PHP on a Linux platform to tackle the same issue. In fact, I have found little of value in any of the CEP IDEs for complex detection. They are “cute” IDEs for processing a type of rule based logic against data. They are wrappers for deterministic logic, for the most part.

    As one of the people who used to be considered at key member of the CEP inner circle, I have no reason nor motive to mislead anyone.

    When I was at TIBCO I briefed the community (keynote) on the first day of our first event that event processing approaches required statistical methods to help end users and that the SQL query-based approaches were not truly CEP. Almost everyone agreed with me (except the query engine vendors) and I was in the (mostly silent) majority. Privately, everyone agreed, even Dr. Luckham. In public, things were different.

    One reason for this is that (IMO), when you move from talking about “CEP” and high level abstractions, to actually using these CEPs to address real complex problems, you will quickly see that these rule and query based (deterministic) IDE approaches only work for a small niche subset of the overall CEP space. They don’t add much value for any of the classes of problems I have worked on over the past 20 years. If they did I would be the first to sing their praises!

    Rule based systems are are useful, but they are not efficient when dealing with complex detection problems that change over time (dynamic v. static). If they were, the CEP vendors would be saying, ‘Tim, use ours and then you can blog otherwise. Let me prove it to you.” but the truth of the matter is that I have installed them and tried them and they are, for the most part, value subtracted. They are IDE toys. It is easier to use open source on Linux/Unix. Perhaps these tools are not well developed on Microsoft platforms? Perhaps that is why CERN uses Linux and open source for Atlas and for their detection problems?

    Your writing is elegant. You have a lot of experience. However, it does not matter how elegant your writing is, that will not change the fact that the current generation of software products on the market that use the term “CEP” are not useful for the majority of CEP classes of problems. That is also why you don’t see much use outside of financial services using them to supplement their trading software. If they worked well (and were a value for the buck) for complex event detection, I would be using them right now.

    The fact of the matter is these query and rule-based IDEs in the CEP space are only useful for a small subset of the CEP space, a very small niche, and pretty much value subtracted elsewhere; unless of course the user cannot write a single line of code, LOL.

    Cheers.

  5. Charles Young says:
    Tuesday, March 2, 2010 at 11:07pm

    It was, of course, me who raised specific technical points in this chain. I note that you still choose not to address them. That, of course, is your prerogative. I believe your post exhibits specific areas of confusion of a technical nature that I won’t rehearse for a third time. I see rule processing as foundational to detecting complex events, but believe that CEP demands a multi-disciplinary approach which is best served by bringing different technologies, techniques and approaches to bear on different aspects of the problem domain. I do not think this is best served by dismissing the role of useful and well-accepted approaches in favour of purported ‘alternatives’ that in fact complement, rather than rival, the rule-based approach.

    I don’t doubt or question the importance of the work you did in 1998 or the insights this has given you. I also work for a living and am involved in building systems that address real-world problems. Some of those systems are large and complex. Many involve rule processing. I am currently working on the architecture and technical design of a national system that will service specific needs of the citizens of a medium-sized nation of approx. 60 million people. This system involves the application of different types of rule processing at several different points within an overall architecture which is, quite literally event-driven. I am involved in the practical implementation of rule processing approaches as a natural part of my job. That job encompasses not only architectural and design work, but also, admittedly less often than was once the case, the more practical side of development and implementation using mainstream programming languages and tools. My interest in CEP stems naturally from my involvement in rule processing. I blog about a particular suite of products and technologies from a specific vendor because, for some years now, I have worked as a salaried employee of a company that specialises in the application of those tools, and which is in partnership with that vendor. Obviously my ‘worldview’ is influenced by the technologies I currently use. I have, however, been working in this industry since the mid 1980’s, so my worldview is fashioned by a somewhat broader perspective. You suggest I have a merely theoretical perspective. You are mistaken. I have a technical perspective, based on practical experience. That is quite a different matter.

    The sad thing, here, is that, like many of your readers, I agree with your repeated claims that CEP is about so much more than event stream processing. Much the same argument applies, incidentally, to other types of rule processing As I keep on saying, I think CEP deserves and needs a multi-disciplinary approach. I don’t think you help to foster that by taking the stance you do against virtually every vendor who dares to claim CEP capability for products with a specific vertical alignment, against whole communities that actually share common and overlapping interests and insights and against those who have tried, in good faith, to give shape and substance to this important area of IT. Instead of providing the well-reasoned, balanced and compelling arguments that would carry people with you, you appear to prefer a far more confrontational approach that is, frankly, not always based on an entirely solid technical foundation. C.f. your post. Confrontation has its place, of course. However, you must expect some kickback.

  6. Peter Lin says:
    Wednesday, March 3, 2010 at 4:10am

    I’ve stated this before, but I’ll repeat myself. Like Charles, I believe a hybrid approach that blends multiple disciplines is the best way to move forward. The trick though is I don’t think anyone knows all those domains with sufficient depth to really show the rest of the world “how to build” a solution with all these different technologies. I don’t know about others, but I like lively debates that challenges me. When I’m out of my comfort zone, I’m forced to re-evaluate and rethink everything. Often times, lively debates gives me a huge kick in the butt and helps me realize I’m missing something. I understand that not everyone likes lively debates, but it does force the reader out of their comfort zone.

    Hopefully one day, some one will figure out how to use all these technologies in an elegant way and teach the rest of the world how to do it.

  7. Tim Bass says:
    Wednesday, March 3, 2010 at 5:10am

    HI Peter and Charles,

    I agree with both of you, obviously, that a multi-disciplinary approach is always best.

    However, I also think I agree with Charles that I have not addressed his technical point.

    I *think* the only technical point I have not addressed by Charles is the paper he provided that showed that researchers at a university claim that rules can be used for uncertainty reasoning. In that paper they showed a belief network, not much different than a Bayesian belief network, and the authors showed how rules can be used as the basis for a belief network.

    Did I miss some other technical point, Charles?

    Regarding that academic paper regarding rules as a belief network; I am not sure what your point it. We could easily move to a type of debate where you provide a link to an academic paper, and I counter with a link to another academic paper. This would evolve into a type of academic poker…. where “I call your two papers on rules-as-belief-networks” and raise you “three papers on Bayesian networks and one paper on neural nets”…. This type of academic paper debate could go on indefinitely. That is why I have not yet addressed that point, because I can counter with papers where “I call your academic paper and raise you three other papers … ” (Edit: Shall we proceed this way?)

    Regarding Charles’ comment on my preference for a “far more confrontational approach”…. you are mistaken (in my opinion) in that area because you do not know the history of this debate, which dates back to the first event processing symposium in 2006. During that time, I presented the concepts of a multi-disciplinary approach to CEP, everyone seeming agreed with. At one time, we had a very strong collaboration going between many of us; however, little progress was make because of all the various conflicts of interest.

    I would be very happy to find people who are willing to collaborate on this topic. I agree with both of you that a multi-disciplinary approach is required to solve complex problems; that is precisely what I briefed to the “event processing community” back in 2006, nearly 4 years ago.

    Regarding confrontation, as you mentioned, confrontation has it’s place, and it does not trouble me at all when people are willing to debate on the technology, not personalities.

    So, Charles… coming back to your technical points, what point would you like for me to address? Let’s start with one, because if you toss 5 at me at once, it does not give me a chance to ask you questions in reply.

    Thanks.

  8. Charles Young says:
    Wednesday, March 3, 2010 at 8:42am

    I raised three points which I labelled a), b) and c) above. I suggest we put c) to one side for the time being. It serves to underline our different perspectives on CEP, but it seems to be that this is based partly on a more fundamental disagreement about the nature of rule processing. So, that leaves a) and b)

    a) I’m claiming that simplicity of form (‘IF…ELSE’) does not imply simplicity of function, and that you appear to have confused the two in your post. You suggest a link between the simplicity of the form of a production rule with a supposed inability to handle complex problems. It appears that you think that rules processing cannot handle complex problems efficiently, and, if I understand you correctly, you seem to suggest that there are also issues with management of large rule sets. My position on all this is as follows:
    1) form is different to function. The simple form of a production rule does not imply an inability to handle complex problem solving requirements using rule sets. Bayes theory itself involves a very simple mathematical form. That’s its beauty.
    2) there is no academic or real-world support for your general dismissal of rule processing as a ‘highly inefficient’ mechanism for complex problem solving.
    3) design and management of large knowledgebases is difficult under any circumstances. Large knowledgbases have inherent complexity. Why would a very large Bayesian belief net be any easier to manage and maintain than a very large rule base?

    b) I’m claiming that Bayes theory offers a complementary, rather than an alternative, approach to rule processing. Bayes theory provides a mechanism for reasoning under uncertainty. This is something we must often do in the context of rule processing, but rule processing is not, in itself, defined as a mechanism for reasoning under uncertainty. I believe a smarter approach is to advocate the use of Bayes theory , and other techniques, alongside rule processing in order to tackle certain classes of problem – i.e., adopting the multi-disciplinary approach we seem to all agree on. I cannot see the relevance, therefore, of the paragraph you wrote that started “However, when you use advanced analytics, like a well-designed Bayesian classifier…”

    Re. the paper, I agree with your latest comment. The abstract of that particular paper does not quite accurately reflect the contents, and appears to suggest some fundamental issue in terms of reasoning under uncertainty in the context of rules processing. In fact, the issue was simply that the approach taken by an ancient expert system called MYCIN exhibited several limitations. The paper advocates a different approach which appears to be a Bayesian one, although curiously this is not stated explicitly. The paper is odd for another reason. At the time of publishing (1987), this issue was already long-settled.

    I’m only aware of the paper because you previously used it to support the claim that “rule-based systems…also have well documented limitations…in the classes of complex problems they can efficiently address” (see http://www.thecepblog.com/2007/07/27/blending-cep-for-rules-and-brms-folks/). You repeat this claim again in this post, saying “However, rules alone are highly inefficient for most classes of (not simple) problems. I have pointed this out a number of times over the years and I think most people ‘get it’” So yes, a form of ‘academic poker’ indeed, except that we appear to be playing the game with just one shared card which you originally laid on the table. And yes, I am absolutely one of the people who don’t ‘get it’ because I can’t see that such a broad and unqualified assertion can possibly be substantiated.

  9. Tim Bass says:
    Wednesday, March 3, 2010 at 9:08am

    Hi Charles,

    So, if I understand you correctly, the main point you would like me to address is scalability of rule-based systems v. statistical classifiers. My claim is that rule-based systems are more complex to manage, and therefore much less efficient than statistical classifiers like Bayesian classifiers? You believe that this claim is unsubstantial and there are no academic papers to support this claim. Is that right? Do I understand you correctly?

    Your other point I should address, if I understand you correctly, is the relationship between rule-based approaches and statistical approaches (like Bayes). You seem to be under the impression that I believe “either/or”. That is not my view and I apologize if I have some across that way. So let me clarify:

    Almost all detection oriented systems (and CEP is a detection oriented technology by definition – detecting opportunities and threats in near-real time), requires considerable logic. On that, I think we agree. I find that in most cases, a combination of rules along with some more advanced analytics (like a statistical classifier) is required. I think you agree with that. However, it seems you are taking issue with my use of the phrase “advanced analytics” and equating that with a statistical classifier, implying that a rule-based approach is not “advanced”… am I understanding you correctly? Is that the root of your concern on this secondary point?

    Finally (for this round), you seem to also have an issue (for me) between the relationship between something like a Bayesian classifier and rule processing, and you only believe they are complimentary and never an alternative to the other. Is that really your view?

    Thanks for clarifying this step-by-step. When I better understand the components and concerns of your technical questions, it will be easier for me to elaborate in my reply (and find the academic references where appropriate so we can enjoy a few hands of academic poker on this topic…)

  10. Charles Young says:
    Wednesday, March 3, 2010 at 11:31am

    My precise point is that, in asserting that rule processing offers, by itself, a ‘highly inefficient’ mechanism for tackling most complex problems, you appear to link this notion to the simplicity of the form of rules. I am saying that form has no bearing on the matter. Simplicity of form does not equate to simplicity of function. Efficiency is not governed by form. Furthermore, I am saying that there is no firm basis is either academic research or practical experience to make the claim you do about efficiency. Rule processing can provide thoroughly efficient mechanisms for tackling a wide range of complex problems.

    I’m sure you can find papers that appear to support this assertion, and much more besides. If you want pursue this route, by all means do so. I referred to a paper that you had used in an earlier post in exactly this way.

  11. Tim Bass says:
    Wednesday, March 3, 2010 at 4:36pm

    Hi Charles,

    For some reason which I have yet to comprehend, you seemingly refuse to answer my questions precisely so I can address them precisely. When you offer a stream of opinions with lots of subjective terms, without being precise, it does not really give me an opportunity to understand what you are staying so I can address your concerns without semantic misunderstandings. You might be using a term differently than I do so we need to be precise in our terminology.

    So, given that you have yet to be precise in my earlier simple questions, yes or no, I will try again and I hope you will be generous enough to answer “yes” or “no” and then elaborate if you choose.

    Let’s start with one simple question. For the record:

    (1) Do you think that rule-based systems, as we are discussing, are a type (part of a class) of expert system(s), yet or no?

    Could you answer this simple question, yes or no, please?

    Based on your answer, I will see if we agree on this simple point (and share that with you) and if we do, then we can go deeper quickly. If we cannot agree on this simple point, then we can explore that.

    Yes, or No? Are the rule-based systems we are discussing a type of expert system?

    Thank You!

  12. Peter Lin says:
    Wednesday, March 3, 2010 at 5:44pm

    Here is my bias take on the term “expert system”. There’s a wide variety of definitions for the term and there’s still disagreement on the exact definition. Here is how I define an expert system to avoid ambiguity. I should state that my definition is a variation of Gary Riley’s expert systems book along with a few other notable books on expert systems.

    An expert system is the schema, data, rules, metarules and metadata in a system, which attempts to capture the “knowledge” of a human domain expert. The schema is the object model or relational model. The data is the operational data that is loaded from some external source, which could be a database or flat file. The rules are if/then statements that capture domain knowledge. Metarules capture information about static and dynamic rules, and can generate/modify rule at runtime. Metadata captures information about the schema and enables the system to modify or extend the model at runtime.

    now that the definition is out of the way, I’d like to point out there are people claiming a “production rule engine” is an expert system. In my book that is wrong and causes a lot of confusion. In my mind, none of the CEP engines out there qualify as expert systems. In fact, I would go one step further and say most of them do not have sufficient functionality. Even if a developer wanted to build an expert system using one of the CEP engines, it’s not possible. Most of them don’t even know how to build expert systems and none of the realize what their product is missing.

    My opinion is that an expert system properly built by an experienced developer can integrate if/then statements with statistical approaches like bayesian and kalman filters. In those situations, the “if” part of the rule may call one or more functions, which utilizes bayesian filters with additional statistical calculation to produce a confidence value. The calculated value is then used in the “then” part of the rule to trigger additional evaluation, which may look at additional variables and factors to improve the accuracy of the result.

    Having said that, very few people have real world experience doing this type of work. I know many of the active practioners and less than a handful really know their stuff. What most “rule consultants” do is simple sequential rules (BPM) using ‘if/then” syntax. There’s no inferencing, and the process is not complex from an algorythmic perspective. The complexity comes from the process, not because of uncertainty or rapid changes.

    In my mind, a lot of the confusion stems from confusion in the domain. There are too many people claiming to be experts in production rules, when they have no clue and no real world experience. Just because a developer worked on 1 or 2 simple bpm projects using a production rule engine, it doesn’t mean they “know jack” about inferencing. A few people like dr. forgy, paul haley, gary riley, ernest friedman-hill and said tabet have breadth and depth in this field.

  13. Tim Bass says:
    Wednesday, March 3, 2010 at 7:13pm

    Hi Peter,

    Thanks for the reply. I look forward to hearing where Charles is coming from when he describes this abstract “rule-based system” that he is referring to, so we can all have a common basis for discussion

    In the meantime, I wanted to comment a bit about this topic in general.

    My past experience writing rules, coding rules, and leading a team of “rule writers” plus my reading in this area, including academic papers on topics like “rule-based reasoning systems’ and similar topics, tends to converge. One key convergence I am referring to is the common principle that all systems, great or small, have advantages and disadvantages. One size does not fit all. After all, I am an EE and studied electrical engineering at the university, not marketing or sales (BTW: I assume you and Charles are also systems engineers like me.).

    The literature is plentiful on papers, slides, presentations, university lecture notes that describe the advantages and disadvantages of rule-based systems, especially rule-based reasoning systems. These advantages and disadvantages are well documented, and they all tend to converge on these principles; and the convergence in theory also confirms what I have seen in practice.

    The next logical step in my reply would be to discuss these advantages and disadvantages. We also need to be precise on terms like “learning systems” v. systems that do not “learn.” Then, we can compare and contrast the advantages of training and learning versus specifying all known possibilities and states of the system (and relate this to our definition of an “expert system”, rule-based systems, rule-base, etc.).

    I think that if we can have a mutual understanding of these core terms and concepts, then we can more easily discuss advantages, disadvantages, efficiencies and inefficiencies. I have an abstract concept in my mind that leads me to my views; and I guess that Charles has a different abstract concept in this mind; and that might be the basis for the gap in our discussion.

    We do need to be precise, and precision is what has been lacking in the CEP world since “day one” and that is why folks like us tend to complain about the lack of “prior art”.

    As a footnote, one of the first slides I presented at the first “CEP meeting” in 2006 was called “A Vocabulary of Confusion” where I illustrated and discussed how this lack of common vocabulary was causing problems. My slide was an evolution (adaptation) of a slide by two top national experts in multi-sensor data fusion, Waltz and Llinus.

  14. Charles Young says:
    Thursday, March 4, 2010 at 6:39am

    I’m looking forward to finding out why you are asking the question about expert systems.

    The term ‘Expert system’ describes an application type. The term ‘rule-based system’ refers to a technology implementation. Most commercial expert systems have been implemented using rule-based technology. It makes no sense to ask if a rule-based system is a type of expert system, so the question has no answer.

  15. Tim Bass says:
    Thursday, March 4, 2010 at 9:47am

    Of course it makes sense Charles. You just do not want to answer simple questions because you have to commit to something and then you cannot “wiggle out” of the precise position.

    You have yet to answer a single question I have asked. Every question I have asked you, you have dodged the answer.

    The reason you do not answer precisely (it seems to me) is that that if you answer my simple questions precisely, then I can precisely answer the question; which you obviously do not want to be “pinned down” because then you cannot “wiggle out”.

    Let’s end this discussion since it is obvious you want to be vague on your answers and are just providing opinions without facts (or being precise in your terms) and you do not want to define the terms you are referring to.

    Thanks for visiting; but it is better we end this circular discussion since you will not answer a single question precisely and in a way it is easy to understand what you are staying.

  16. Tim Bass says:
    Thursday, March 4, 2010 at 10:02am

    PS:

    I will write a blog soon called something like:

    “Why Rule-Based Systems Are Resource Intensive and Inefficient”

    I will provide helpful references, all of which are readily available on the Internet.

    However, I doubt that posting facts and prior art will quell the “religious about rule” nay-sayers, LOL.