John R. Boyd
John R. Boyd

For those of you who are interested in complex event processing and decision theory, and in particular, deduction, analysis, and differentiation vis-a-vis induction, synthesis, and integration, I highly recommend you read (don’t hesitate one second) a paper written more than 30 years ago.

The paper is called Destruction and Creation by John R. Boyd, published on September 3, 1976 and is, arguably, the best eight pages you can read.   It is amazing that a mere handful of pages written 30 years ago can say more about “complex event processing” than the vast majority of literature we see today.

In a few insightful pages, John Boyd eloquently explains the need for decisions.  He synthesizes amazing works-of-science-and-art like Kurt Godel’s World of Mathematics and Logic, Werner Heisenberg’s Indeterminacy Principle with the concepts of confusion and disorder from Entropy and the Second Law of Thermodynamics.   Here is one example out of his paper:

[W]henever we attempt to do work or take action inside such a system—a concept and its match-up with reality—we should anticipate an increase in entropy, hence an increase in confusion and disorder. Naturally, this means we cannot determine the character or nature (consistency) of such a system within itself, since the system is moving irreversibly toward a higher, yet unknown, state of confusion and disorder.

So, when you hear folks complaining about the “confusion” in CEP, please understand that confusion and disorder are natural products of our attempt to do work inside of a body of knowledge.   Also keep in mind that the process of creation and destruction of ideas, concepts, notions and abstractions are the very fabric of how we must work to progress the state-of-the-art.

In fact, the more we resist the natural creative and destructive processes, the more we inhibit progress and change.  So don’t create draft vocabularies for event processing and then refuse to destroy it with the same ease we drafted it!   Remember, we are constantly taking specific concepts and integrating them, creating disorder and confusion while at the same time, we are tearing apart the same abstractions trying to put an end to the confusion we created.

However, we humans hopeless love to cling to our ideas, concepts and we get wrapped up in “who invented this” and “I said this first” and “we defined this already” etc. ad infinitum.   Opher Etzion and others want to paint me as “negative” because I want to completely change the direction of much of the current work in event processing.  Mark Palmer, now of Streambase, has said I am trying to “destroy CEP” and questions why I “would do such a think to something I love so much.”  (Mark even has asked me, endearingly, “What is wrong with you, Tim?!”)

Actually, it is because I love (complex) event processing so much that I am willing to tear the current concepts apart, because until we do, we will be impeding necessary progress and change.   This is simply how nature intended it to be and nobody can explain this natural process better than John Boyd.

It is a must read.  One final quote (the abstract) :

To comprehend and cope with our environment we develop mental patterns or concepts of meaning. The purpose of this paper is to sketch out how we destroy and create these patterns to permit us to both shape and be shaped by a changing environment. In this sense, the discussion also literally shows why we cannot avoid this kind of activity if we intend to survive on our own terms. The activity is dialectic in nature generating both disorder and order that emerges as a changing and expanding universe of mental concepts matched to a changing and expanding universe of observed reality. – John R. Boyd.

9 COMMENTS

  1. I for one appreciate the frank opinions on this blog. Although I understand people have to tow the company line, it doesn’t help the domain mature. I don’t know if others feel this way, but my bias perspective is the users have to push back. If users never speak up about the limitations and deficiencies, products might not improve.

    Rather than point out opinions from users are negative, I would think it’s more productive to figure out how to improve their products. Shooting the messenger doesn’t negate the message and only makes it look worse. I hope the CEP industry takes a hard look and figures out how to expand and improve their products.

  2. Hi Tim. Thanks for mentioning me by name, however please quote me correctly. I did not use the term “negative”, on the contrary, I have given you an advice, which is true for everybody, that messages have higher probability in getting traction if they are phrased in a positive way (the term negative is your own inference) , One can certainly makes suggestions about things are missing in products, direction that the entire community should follow etc.. however, I have an old guy, and I have learned from my own experience that when the style of how things are said becomes an issue by itself, the message (which may be smart) may get lost,

    A comment to Peter Lin — I fail to see how giving an advice on style relates to the company party line.

    Just a comment about the “what is CEP” campaign — while I prefer not to use the term CEP, and my company IBM does not use it either (so again no party line involved here), I also not very bothered by it — there are many names that came from historical reasons and are not the best scientific names I would call an area — anyone knows how the term “operations research” has been coined ? answer: unification of two departments in the UK RAF in world war II. Does it provide good reflection of what this discipline do ? – no.
    Does an enterprise that use OR techniques get more or less value due to the name ? – no. You claim to speak on behalf of customers, well – my impression from customers that don’t really care if a technology is labeled CEP, ESP, EPP, or Apazacta. If a technology called Apazacta has real business value, the customer will buy it… The properties of a technology is what is important, and focusing the discussion on substance and not on labelling will be more beneficial to the customer community…. I guess that we’ll have to agree to disagree on that.

    stay well,

    Opher

  3. Hi Opher,

    I agree with you that customers with problems to solve generally don’t care very much about labels CEP; however, just because they don’t care much about it does not justify folks using the terms wildly incorrectly, as we see today.

    And can we blame them?

    The EPTS glossary is so far off target that the best way to improve it is to destroy it and create another one! The term “complex event” is a nearly a joke (technically), as it “complex event processing”.

    Let’s destroy these barriers to progress and create something new that is correct. We can’t make progress if we create drafts that are wrong, edited and controlled by a handful of people who think they are the “authority”, and then refuse to destroy it and create something new.

    If terminology is not important, then why have a glossary that people often refer to like a Bible? Destroy it before it is too late, LOL.

    Yours faithfully, Tim

  4. Hi TIm,

    The editors of the glossary can speak by themselves and I don’t need to defend them; I encourage you and anybody else to make suggestions and try to argue about the definition of the term “complex event” or any other term – anything can be changed if there is a good reason to do it, and I have full confident in the glossary editors (who don’t need endorsement from either me or you to be considered as authorities in the community).
    It is totally valid to have difference of opinion, and have discussions., However – what I am trying is to keep the level of discussion civilized.
    There are more effective way to try and convince people about the definition of a certain term than Showing disrespect to people personally (“authorities”) and to their work (“joke”) .

    cheers,

    Opher

  5. Dear Opher,

    Complex events and complex event processing has been around for a long time, much longer that Professor Luckham’s book.

    Just because he coined a phrase does not mean that he is an authority on every facet of event processing or complex event processing. Here is my post in reply to David on his site:

    “Hi David,

    You have stated publicly that CEP has been around much longer than your book, it was just called something else. There is much more prior art and authority to “complex event processing” than your book, I am sure you would agree. Your book did not mention much of the prior art on event fusion, sensor fusion, detection theory, prior distributed event processing architectures, control theory, detection theory, and so many many related fields.”

    Yours sincerely, Tim

  6. Making style suggestions is valid in my mind, but I see it as a form of censorship. People censor themselves every day, so it’s just a fact of life. For me, given a choice between a politely worded critique and a blunt unfiltered comment, I prefer blunt comments. That’s a personal preference and I understand not everyone responds well to it.

    I agree that making style suggestions doesn’t necessarily mean towing the company line. Not everyone working for company is able to walk that line.

  7. Hi Peter,

    Thank you for your supportive words.

    Actually and factually, I told the CEP/EP community in March of 2006 very politely the same things I have been saying now. However, after 3 years of watching the same crowd ignore the prior-art, fail to reference the decades of background work, change and position technical terms for the sake of marketing, while at the same time see the state-of-the-art moving backwards, not forward, my comments are becoming more sharp.

    Even as one of the first committee co-chairs for a future EPTS working group, and as a founding member, Opher Etzion has completely discounted all my suggestions how to build a better organization and has surrounded himself with people that does his bidding or sees things in the same way.

    So, yes, I agree with Peter Lin, Opher’s comments about “negative” and “style” are more about attempts to both discredit and censor. In fact, I have not been completely honest about the content of Profession Luckham’s book, I have been very nice and self-censoring myself, because if I published my true thoughts about the lack of mentioning prior art, references and other facets of the long history of event processing, my comments would be much harsher, more sharper and more factual.

    I have been contemplating ending my self-censorship, however, and write an honest, non-censored critique of complex event processing (the book and the marketing), from the point of view of the work and people who were not referenced and the prior art that was ignored in “The Power of Events”.

    In other words, I have been very generous to David Luckham and the CEP/EP community.

    Yours faithfully, Tim

Comments are closed.