The Complex Event Processing Blog

Lessons Learned from High Tower’s Demise

In November 2008, Aliso Viejo-based High Tower Software, a venture-backed developer of security, compliance, and log management software, shut down.   Like many of our “CEP/ESP vendors”, High Tower orchestrated numerous “awards” for their security information and event management (SIEM) software, However, these fluffy marketing awards were not enough to keep HT from a nose dive.

High Tower’s focus was IT security products that supported real-time visibility, alerts, and reports on network threats and organizational policy violations with proprietary analytics they called MetaRules™.  I took a close look at High Tower’s “MetaRules” when I was working for TIBCO a few years ago.   Basically, the “MetaRules engine” was a very simple rule-base that had very little, to no, value added analytics or advanced correlation techniques.  The High Tower architecture was build on a simple centralized, trivial rule-based architecture, so as a distributed-network systems architect focused on CEP at the time, I advised High Tower executives that their software architecture was not scalable and their analytics were pale in comparison to TIBCO’s flagship event processing product.

My advice was not well received, as I think High Tower executives were expecting me to heap praise upon their technology.  In fact, their rule-based approach was far too simple for most information security challenges.   The High Tower folks booted me out the door, so to speak, because I told them the painful truth about their technology, based on decades my of operational experience in network and security systems engineering and management.

During this same time period, I have also advised the self-styled CEP and ESP vendors.  These centralized rule-based approaches, admittedly better than High Tower’s approach, are still years away from being capable of detecting anything but the most simple scenarios in complex security and network management problems.  Instead of addressing these serious shortfalls, most of these vendors are in denial, just like High Tower was “offended” after I spend a few days analyzing their software architecture, rules engine, and overall approach.  ESP vendors get excited when someone posts a trivial situation-detection problem and argue that it is “not really event processing” if their solution cannot easily process it.

There are very serious lessons to be learned from High Tower’s fall.  First of all, listen and learn.  I hope that in 2009 vendors will not dismiss my advice when I explain to them how routing and scheduling is not complex event processing.    When we look at the blogs and logs about complex event processing, the focus is still SQL-based stream query processing in sliding time windows.  This is a niche area which addresses less than 10% (pick a different number if it makes you happy) of real-time detection oriented problems.  Like High Tower, vendors in the CEP/ESP/EP space must evolve, or die.

Second, do not base your solutions on centralized approaches.  Most stream processing engines we see on the market centralized client-server architectures.  Events are pumped into a centeralized stream processor and some simple rules are applied against the stream.  This approach is acceptable if you are  viewing these engines as a type of edge device in a large event processing scenarios, filtering, aggregating and routing.  However, in a more general approach, you need collaboration and cooperation from multiple agents.  Ironically, these CEP/ESP stream processing engines do not even make a good scheduling in a complex distributed problem-solving, agent-based architecture.

Finally, incorporate advanced analytics sooner than later.  Rules are limited by a number of factors, the same limitations that are well known in expert-systems.   Advanced detection requires a number of sophisticated analytics.  Until this happens, the CEP/ESP engines on the market today will continue to push routing, orchestration and simple detection solutions, while a the same time, avoiding complex event processing detection scenarios.

The clock is ticking on CEP and the alarms bells are not very far away.   Companies who do not listen and adapt will find themselves in a similar situation as High Tower and their investors.