Disadvantages of Rule-Based Systems (Part 1)

Posted on 03/05/10 8 Comments

In Orwellian Event Processing the discussion moved away from my original intent, which was primarily to discuss the vendor-state-of-denial regarding the prior art for processing complex events, and gravitated toward a discussion on the “inefficiencies” of rule-based systems.  I was surprised learn that there are professionals who believe that there is no basis in fact (or supporting literature) for saying that rule-based systems are inefficient.

The reason I was surprised at this was that I have experienced these disadvantages in operational network and security management systems for years (where it is nearly impossible to specific all possible combination of text in a string or strings and process them without false positives and negatives).  I have known teams of people writing rules to update a rule-based approach to identifying hostile email bombs.   I have also experienced the same type of problem with other common classification problems such as classifying text as “spam” or classifying text as “code” or “prose” or classifying log file entries as “friendly” or “foe”.

In fact, it was after experiencing these limitations that I began research into this topic.  One place to start is to simply Google the phrase, “disadvantages of rule-based systems where, at the time of this post, this Google search resulted in:

Results 110 of about 11,800 for “disadvantages of rule-based systems”. (0.12 seconds)

Puzzled by the seemingly religious defense of rules-based systems by folks who work with these systems (as if there were no documented disadvantages anywhere on the planet) I receive an email yesterday that helps explain it, maybe.  Here is an excerpt from that private message (keeping the identity of the author private):

In the past I’ve had flame wars with the Drools and other vendors over these topics. Too many people in the BRMS space hide these challenges and hate it when people bring it up.

This jives with my experience since I was involved with CEP.   Regardless of my approach to this discussion, there are people out there that believe in rules so much, they lose there objectivity when we discuss the disadvantages.  I never had this problem prior to working with CEP.

Moving forward, we “engineers” know that all systems (Yes ALL) have strengths and weaknesses, advantages and disadvantages.  In fact, the strength of a system often defines it weakness and it’s weakness defines it strength.  Advantages and disadvantages of a system are two sides of the same coin, one cannot exist without the other.    So without diving into Zen or deep philosophy on “why a strong tree breaks in a storm and dies, where a soft blade of grass bends with the wind only to spring back the next day” however, “you don’t build your house out of grass“, I will gently take us to Disadvantages of Rule-Based Systems (Part 2) in the next post on this topic.

The disadvantages (and advantages) of encoding knowledge as rules and using this system to process complicated tasks has been firmly established for many years.   Two of the key disadvantages are really a type of “common sense” where we don’t need 11,800 search results to find the answer, we just need to work with rule-based systems applied to complex, dynamic problems and we will quickly see the issues.

We could Google and review the numerous slides, lecture notes, papers and so forth on the advantages and disadvantages of rule-based systems, and provide links and quote to the specifics, but I suspect that the debate would still rage-on.   For me rules-based systems are just one of many tools in a very large software tool box.

I like software rules.   Few, if any, computing systems would work without “IF-THEN-ELSE” logic,  whether declarative or programmatic.  On the other hand, I always find myself using rules as mainly the “front end logic”  and the “back end logic” when working on complex detection problems. The heart of the detection is (often) a statistical classifier which can “learn” and is “trainable” without writing more rules.   Does that mean rules are “bad”.  No!  It means that system architects need more than rules in their toolbox to deal with complex classes of detection oriented problems.

In closing, I question myself as to why I am writing this post at all.  To any objective person, it is obvious that there are advantages and disadvantages to using the various tools in the systems architect’s toolbox.   I “guess” the reason I write on this is that I remain discouraged at the press releases and blog posts from the “event processing community” who have seemingly defined “complex event processing” as a type of metaphor for rule-based and query-based processing, seemingly marginalizing all the other well established (and very important) tools in the toolbox of the systems architect.

Share and Enjoy:
  • Print
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks

6 Comments

  1. Peter Lin says:
    Friday, March 5, 2010 at 6:30pm

    Managing rules is not a trivial task whether the environment is static like business rules or dynamic like operational network security. In the hands of an experienced rule developer, it “can” work and provide an usable solution. Keeping in mind “usable” doesn’t mean pain free, easy or self-managing.

    There are thousands of papers on the horrors of rule management for a simple reason. The developers didn’t bother to learn how to use rules. I’m willing to bet 90% wrote the rules as if they’re writing C, Java, C#. There are literally hundreds of horror stories going back to late 80′s.

    The real downside of using a production rule engine and expert system approach is finding qualified developers that know what they’re doing. A large majority of the “rule consultants” are complete hacks. My tendency is to warn customers about these challenges before they use a BRMS. Most are thankful for the information. The vendors on the other hand work very hard to hide these pitfalls.

    my bias opinion.

  2. Greg Barton says:
    Friday, March 5, 2010 at 7:38pm

    In my experience the reluctance to discuss the disadvantages of rules based systems comes from the fact that most criticism the rules based practitioner encounters comes from ignorant sources. Easily 90% of the time when I’ve encountered someone who says “I tried rules and it was sooooooo sloooooow” they’ve implemented their rules in one of two ways:

    1) if (true) then [a bunch of procedural code]”
    2) if [cartesian join the whole working memory] then [complicated RHS that modifies a bunch of stuff]

    So the population of folks who actually have something constructive to say is generally overwhelmed by those criticizing the technology from a point of ignorance. Obviously you’re not coming from that position of criticism, but I’m sure you understand the toll of having to deal with it day in and day out. One tends to become a bit defensive.

  3. Tim Bass says:
    Friday, March 5, 2010 at 7:53pm

    Hi Greg,

    Thanks for stopping by and commenting.

    My experience has been that the folks who are not objective on the side of the rules discussion are not ignorant, but they are narrow in their view. Smart people with narrow views make stronger arguments than ignorant people. Ignorant people cannot formulate well written logical fallacy. Very clever people are good at logical fallacy.

    In the CEP forums, there is one person who has argued with me so much that every thing I say is wrong (and all my family members are bad too, so he says… haha!). Then, after six or nine months pass he is claiming all this new genius (which is what I told him 6 to 9 months ago) and now he argues about the next domain of knowledge he has yet to acquire!

    I don’t understand the zeal about rule-based systems and rules, personally. What makes them so “sexy” that people defend them as if they are a belief system that must be taken as fact on faith? All systems have good and bad.

    Currently we are working on a classifier to determine is there is programming code in text (C, PHP, shell script) and we would never attempt this with rules, never! We would be writing rules forever!

  4. Greg Barton says:
    Friday, March 5, 2010 at 8:46pm

    I guess what makes them sexy is that business folk are somewhat sold on them. It appeals to the top down mentality, which is generally held by those at the top. :) It’s the “you can be in control of your business because business people will write the code” meme, which as we know is more than myth.

    This goes the opposite direction with the bottom up approaches. “You mean that it figures things out without a human telling it what to do? Isn’t that DANGEROUS? And if it does the wrong thing who can I fire / sue / punish horribly??” It’s hard enough trying to sell the complexity of rules to your average vice president. Try telling them “the system figures it out all on it’s own.” Oi!

    But ya, it’s not just the folks at the top who drink the rules kool aid. On one rules project I tried introducing some fuzzy knowledge engineering techniques: letting domain experts classify the similarity of concepts using a graph, putting similar concepts closer to each other. The project lead (who will probably read this eventually and rip me a new one… :P ) said that my approach was “just an algorithm” while the rules were “real AI.” Makes me chuckle to this day. :)

    That mentality is really tied to one thing Peter mentioned in the Orwellian thread, and is a classic problem in the AI world: the top down folks and the bottom up don’t get along, and few people are really versed in both worlds. I encountered this myself a few years ago when I mulled getting a PhD. My interest was (and is still) in applying evolutionary computational techniques to create rete networks. (And I found a neuroevolutionary technique that could be adapted to do that.) The number of people who could sponsor doctoral work like that is quite small.

    It strikes me as partly a cultural problem that’s driven by the way people’s brains work: you’re generally drawn to one discipline based on how you instinctively approach problem solving. (To make a gross generality.) And once you wade deep into one side of the divide there’s cognitive as well as cultural barriers hampering you from wading to the other side.

  5. Tim Bass says:
    Friday, March 5, 2010 at 9:23pm

    Hi Greg,

    That was really insightful. Thanks for sharing. I learned something both relevant and useful from your experience. You are right. Rules are instinctively easier for the “management types with the budget” to understand than a system that is described in more exotic terms like “learning” and “training”.

    I can hear the CFO now…

    Reminds me when I first tried to describe pub-sub messaging to senior government officials….. we get labeled a “geek” very quickly in that scenario.

    Personally, on many of the classes of problems I’ve been working on lately, I much prefer to cut-and-paste a false negative to my “training Interface” than logging in a writing and testing a rule, which may have unforeseen consequences on other existing rules in a large rule-base. (I am lazy…)

    I also think this is why “CEP vendors” in the query space are uncomfortable. The more complex rule management becomes, the more they can promote their IDE. If you take the complexity out of the user interface, and leave the complexity to the back-end (learning algorithm, classifier, etc), it becomes a harder sell.

    Your post reminded me of many of the challenges I faced when I worked as the most senior technical consulting level to the USAF and DOD years ago, and has brought forward a lot of memories.

    Thank you.

  6. Peter Lin says:
    Saturday, March 6, 2010 at 2:58am

    Here is another example. Some times the business people want to validate the business rules have been implemented correctly. The problem is, after a knowledge engineer reworks it to be efficient, the business people can’t understand it. It no longer looks like the business rules they wrote down, which leads them to ask “how do I know that’s correct?”

    I’ve seen this first hand translating lengthy compliance rules to use data driven rule techniques. The business users had a very hard time understanding a single rule could be used for any or all portfolio. In their mind, the rule had to be “hard coded” for them to verify it is correct. Even after I explained, the approach reduced 50-100K rules down to 300 they still complained “it’s too hard to understand”. I got around that by building test cases and showing it produced the result they desired. Sometimes that battle can’t be won and developers end up writing rules in a stupid way to satisfy non-tech savvy business users.

    peter

2 Trackbacks

  1. [...] Disadvantages of Rule-Based Systems (Part 1) | Cyberstrategics [...]

  2. [...] Disadvantages of Rule-Based Systems (Part 1) | Cyberstrategics [...]