Cyberstrategics Complex Event Processing Blog

In November 2008, Aliso Viejo-based High Tower Software, a venture-backed developer of security, compliance, and log management software, shut down.   Like many of our “CEP/ESP vendors”, High Tower orchestrated numerous “awards” for their security event and information management (SIEM) software, However, these fluffy marketing awards were not enough to keep HT from a nose dive.

High Tower’s focus was IT security products that supported real-time visibility, alerts, and reports on network threats and organizational policy violations with proprietary analytics the called MetaRules™  I took a close look at High Tower’s “MetaRules” when I was working for TIBCO a few years ago.   Basically, the “MetaRules engine” was a very simple rule-base that had very little, to no, value added analytics or advanced correlation techniques.  The High Tower architecture was build on a simple centralized, trivial rule-based architecture, so as a distributed-network systems architect focused on CEP at the time, I advised High Tower executives that their software architecture was not scaleable and their analytics were pale in comparison to TIBCO’s flagship event processing product.

My advise was not well received, as I think High Tower executives were expecting me to heap praise upon their technology.  In fact, their rule-based approach was far to simple for most information security challenges.   The High Tower folks booted me out the door, so to speak, because I told them the painful truth about their technology, based on decades my of operational experience in network and security systems engineering and management.

During this same time period, I have also advised the self-styled CEP and ESP vendors.  These centeralized rule-based approaches, admittedly better than High Tower’s approach, are still years away from being capable of detecting anything but the most simple scenarios in complex security and network management problems.  Instead of addressing these serious shortfalls, most of these vendors are in denial, just like High Tower was after I spend a few days analyzing their software architecture, rules engine, and overall approach.  They get excited when someone posts a trivial situation-detection problem and argue that it is “not really event processing”, if their solution cannot easily process it.

There are very serious lessons to be learned from High Tower’s fall.  First of all, listen and learn.  I hope that in 2009 vendors will not dismiss my advise when I explain to them how routing and scheduling is not complex event processing.    When we look at the blogs and logs about complex event processing, the focus is still SQL-based stream query processing in sliding time windows.  This is a niche area which addresses less than 10% (pick a different number if it makes you happy) of real-time detection oriented problems.  Like High Tower, vendors in the CEP/ESP/EP space must evolve, or die.

Second, do not base your solutions on centralized approaches.  Most stream processing engines we see on the market centralized client-server architectures.  Events are pumped into a centeralized stream processor and some simple rules are applied against the stream.  This approach is acceptable if you are  viewing these engines as a type of edge device in a large event processing scenarios, filtering, aggregating and routing.  However, in a more general approach, you need collaboration and cooperation from multiple agents.  Ironically, these CEP/ESP stream processing engines do not even make a good scheduling in a complex distributed problem-solving, agent-based architecture.

Finally, incorporate advanced analytics sooner than later.  Rules are limited by a number of factors, the same limitations that are well known in expert-systems.   Advanced detection requires a number of sophisticated analytics.  Until this happens, the CEP/ESP engines on the market today will continue to push routing, orchestration and simple detection solutions, while a the same time, avoiding complex event processing detection scenarios.

The clock is ticking on CEP and the alarms bells are not very far away.   Companies who do not listen and adapt, will find themselves in a similar situation as High Tower and their investors.

Filed under: Advanced Event Processing, Agents, Analytics, CEP News and Events, Complex Event Processing, Cybersecurity, Event Processing, Event Stream Processing, Situation Models, Systems Engineering, Use Cases | 1 Comment »

I was pleasantly surprised when I read Seth Grime’s Complex Event Processing as a Marketing Device. Normally when I read CEP-related posts by Seth I tend to grimace more than just a bit, as Seth tends to write about event stream processing people (the “SQL-ish continuous query folks”) and products versus the large vision of event processing or the challenges of actually doing complex event processing.  There may be some commerical influences, as I believe Seth recently authored, or coauthored, a marketing paper for Coral8.

Today I read some interesting opinions from Seth (Edit: via Hans Gilde) that were more objective that some of his prior “I drank the Kool-Aid!” posts on stream processing; and I am quoting below.

The market for the brand CEP was always destined to be disappointing. How many of these buzzword techno-brands have ever worked out well? They are the things that go through hype cycles, are discussed by analysts and are often believed to be very “strategic.” And we all know that 80% of that stuff is either total BS or exaggerated for career promotion. From a finance perspective, we are also very aware of how to exploit and then abandon the fad built around such a buzzword. The capital market has gone through an upswing in capital for fad CEP, and now will experience a the abandonment phase. Which will be compounded by the overall capital market problems (duh).

I completely agree with this and am very happy to see others are starting to be “truth speakers” versus channels for the overhyped marketing we have seen for quite some time.  Honestly (and sadly),  I was beginning to wonder if  “software marketing” and “fraud” were synonyms!

Seth (Edit: actually Hans Gilde) also opined:

Over the next couple of years this market can support maybe three big vendors, one small vendor and maybe a profitable but small operation founded around a fundamentally OSS offering. This leaves about 20 small vendors and one or two big ones out of luck. And considering the fragmentation, I think it’s unlikely for a vendor that is now still small to become big. All that’s left is the knife fight among the small vendors for that one surviving spot and the slow attrition that will cause one or two big players to abandon their product line.

This is true only if these vendors do not evolve beyond continous query, stream processing engines (streams, feeds and speeds), and there has been no indication that they will evolve.   The problem is that we don’t really have any CEP vendors calling themselves “CEP vendors”.  (Hint: All the folks doing CEP are actually writing code and solving complex problems; and none of these companies are selling continous query, stream processing engines.  They are working hard to to solve complex problems with the appropriate complex analytics and algorithims.)

Seth (Edit: actually Hans Gilde) concludes:

There never was a “real CEP” approach, just a of “CEP brand” approaches.

Actually, there was first an “ESP brand of approaches” and, at least, folks were truth tellers and not claiming to be CEP experts.  However, for some strange reason, when we informed everyone that ESP was a useful, but incomplete subset of CEP, the same ESP vendors seemed to take offense “to being a part of the solution”, and they had to be “the whole solution”.   This was a big mistake for CEP and it has been the marketing guys who have dug the grave they are starting to see (a glimpse of).

As Abraham Lincoln said,

You can fool some of the people all of the time, and all of the people some of the time, but you can not fool all of the people all of the time.

When the evangelists and marketing folks started to say that a complex event is simply an abstraction of two or more events, any abstract, the writing was on the wall…..

You cannot fool all of the people all of the time.

---

Edited: Seth Grimes was quoting Hans Gilde, said Marc Adler. Thanks Marc!

Filed under: Analytics, CEP News and Events, Complex Event Processing, Coral8, Event Processing, Event Stream Processing | 2 Comments »

Here is my draft list of the Top Ten Cybersecurity Threats for 2009.  Your comments are greatly appreciated.  I will publish the final list later this month, based on comments received.

— Constant negative news reporting and adverse analysis undermining public and business confidence in leadership, business management and economic recovery efforts.

— Criminal manipulation, fraud and subversion of financial markets by opportunists and competitors.

— Identity masquerading to abuse, attack, blackmail, bully, extort, terrorize or molest others.

— Criminal fraud by password and identity theft via phishing, pharming, spyware, malware and theft of hardware.

— Criminal use of botnets and botnet-like technologies, for example email (marketing and rumor) spam and denial-of-service attacks.

— Criminal exploitation of social networks.

— Criminal use of cloud computing and software-as-a-service infrastructures for cyberattacks and spam.

— Spying and theft of data by governments, industry, terrorists and other criminals.

— Sabotage, theft and other attacks by disgruntled employees and insiders.

— Natural disasters, accidents, errors and unintended consequences without malicious intent.

Filed under: Complex Event Processing, Cybersecurity, Threats and Vulnerabilities | 14 Comments »

Reposted (adapted) from The Complex Events Forum:

David Luckham:  Do you expect the market to expand?

No. The entire software market, for the most part, will have a very tough year in 2009, due to a very serious global economic crisis. Firms that have “bet the farm” on capital markets will suffer even more, as these firms are in a deep recession and IT spending always suffers in these environments. We might see some “smoke-and-mirrors” in CEP marketing-land, claiming an expanding market, but these claims will be unfounded and unsupported. As a side note, the largest software companies (in the CEP space) tend to bundle their CEP sales into enterprise software license deals, so it is nearly impossible to correlate actual CEP software sales and usage with what we read from these large vendors (and their reports are the lion’s share of the market).

David Luckham: Does the CEP technology currently offered by products on the market need improvement? If so, what improvements do you want to see?

Of course they need improvements! If the current generation products on the market actually could detect (difficult to detect) complex events in real-time with high confidence, the market would be much, much larger than it is today. I have discussed this in great detail in 2008, and will spare you all the pain and frustration of reading it again. Cheers!

David Luckham: Are there problems out there that customers are willing to pay to solve, but the current generation of CEP products cannot solve? What are they?

Yes of course there are plenty of “complex event” classes of (difficult and complex) detection-oriented problems people are willing to pay for. As I said before, if the software on the market actually did true “complex event processing” (not a watered down version of “everything is marketable as CEP” as we have seen in 2008) then software would be selling like cold water in the hot, dry desert. This is not rocket science. Proprietary stream processing (continuous query) engines have limited commercial value in the overall market, and the market is overly crowded with these vendors! This situation is unsustainable, especially in this (terrible) economy. As someone else opined, these many “doing the same thing” (ESP) vendors cannot all survive in such a negative business climate with so little new actual value-added technical (detection) capability.

David Luckham: Has the time come to talk about standards or interoperability of CEP products?

Software is already interoperable at the message / transport layer, and messaging is rapidly becoming a service-oriented commodity (see my recent post on Amazon SQS). Interoperability needs to happen at the “complex event modelling layer”, and this is not likely to happen in the foreseeable future, since we have so few “CEP vendors” in this space actually doing what I would call CEP. Most vendors are focused on routing, scheduling and simple (not complex) detection scenarios, a software market which is not sustainable, in my opinion.

David Luckham: And discuss any other CEP questions that come to mind!

Yes! But, I’m busy working on a few cybersecurity predictions and community (Web 2.0) software tasks for a major forum, so “that’s all for now on CEP!”

Filed under: CEP News and Events, Complex Event, Complex Event Processing, Event Processing, Standards, Systems Engineering | No Comments »

Does Amazon SQS and other “messaging as a service” applications mean that companies can start to think about reducing their ongoing expenses of licensed or hosted messaging systems?

According to Amazon, Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.

Amazon SQS works by exposing Amazon’s web-scale messaging infrastructure as a web service. Any computer on the Internet can add or read messages without any installed software or special firewall configurations. Components of applications using Amazon SQS can run independently, and do not need to be on the same network, developed with the same technologies, or running at the same time.

Functionality:

• Developers can create an unlimited number of Amazon SQS queues, each of which can send and receive an unlimited number of messages.
• New messages can be added to a queue at any time. The message body can contain up to 8 KB of text in any format.
• A computer can check a queue at any time for messages waiting to be read.
• A message is “locked” while a computer is processing it, keeping other computers from trying to process it simultaneously. If processing fails, the lock will expire and the message will again be available.
• Messages can be retained in queues for up to 4 days.
• Developers can access Amazon SQS through standards-based SOAP and Query interfaces designed to work with any Internet-development toolkit.

It may be possible that some software companies selling messaging and SOA software will find they have been “out-serviced” by companies offering cloud-messaging services!

Filed under: CEP News and Events, Complex Event Processing, EAI ESB & SOA, EDA, Event Processing, Event-Driven Architecture, Message-Oriented Middleware, Virtualization | 1 Comment »

Soon I will publish my The Top Ten Cybersecurity Threats for 2009. However, before doing so, let’s take a look at some interesting aspects of The Top Ten Cybersecurity Threats for 2008.  In my cybersecurity threat list for 2008, I mentioned:

— Criminal manipulation and subversion of financial markets.

What we observed in 2008 was much more than criminal threats. Now we are seeing both intended and unintended consequences of cyberspace.   For example, analysts from competing banks use the Internet to spread “doom and gloom” rumors and flawed analysis to do serious harm to their competitors.

In my post Cyberattack! Manipulation and Subversion of Financial Markets! we discussed the situation of a direct competitor to E*Trade Bank, Citigroup, using the power of cyberspace, rumors and (mis)information to manipulate investor confidence in  E*Trade (ETFC).  This might have not been such an eyebrow raising event if the analyst rumor, released like a bomb in cyberspace, was by a disinterested third party.  The “cybernews bomb” was released by a direct competitor with their own subprime balance sheet problems.  In reality, Citigroup came closer to bankruptcy than E*Trade!

This is a direct abuse of cyberspace and a purposeful, malicious action that can threaten every business in today’s modern networked, connected world.

Moreover, potential more harmful threats are the unintended, not directly malicious “doom and gloom” reports, analysts opinions and news stories, where the entire cyberworld is full of “doom and gloom”, driving the world’s global economics into a downward spiral.

In other words, the biggest looming threat to cyberspace is not the low level hacks and attacks that IT professional often discuss.   The biggest threat is cyberspace itself and how malicious rumors in cyberspace can destroy confidence in sound businesses in milliseconds.  In addition, cyberspace “doom and gloom” has taken on a life of its own, much like a global personality. Unfortunately, we don’t have “cyberdrugs” to treat “doom and gloom” global information-based depression.

Many folks are worried that the global economy will be even worse in 2009.   Never in our history have we had such a global economic downturn combined with global cyberspace “doom and gloom” messaging pumped into our news readers and brains 24 hours a day, 365 days a year, globally and instantaneously.

Indeed, cyberspace itself has become a serious threat, perhaps the greatest threat for 2009.

Filed under: Artificial Intelligence, CEP News and Events, Complex Event, Complex Event Processing, Cybersecurity, Financial Services, Threats and Vulnerabilities, Use Cases | 3 Comments »

Following up to my earlier posts on Event Processing Technical Society (EPTS) related topics, I am proposing a draft reorganization of the EPTS Steering Committee. The current EPTS Steering Committee was structured as follows:

#	Industry	Primary EPTS-Related Technology	Comments
1	Software Company	Event Stream Processing / Continuous Query	Vendor
2	Software Company	Event Stream Processing / Continuous Query	Vendor
3	Software Company	Event Stream Processing / Continuous Query	Vendor
4	Software Company	Event Stream Processing / Continuous Query	Vendor
5	Software Company	Event Stream Processing / Continuous Query	Vendor
6	Software Company	Rete Rules Engine + Continue Query Engine	Vendor
7	Technology Analysis / Convention Host	Research Notes, Presentations, White Papers	Vendor
8	Independent Consultant	CEP Promotion, CEP Web Site Sponsorship, CEP Evangelism	Vendor
9	To Be Confirmed	To Be Confirmed	Vendor
10	To Be Confirmed	To Be Confirmed	Vendor

Discussion:

The current EPTS Steering Committee (SC) does not represent the art-and-science of event processing. All SC members are vendors and all but two of the SC members are software vendors, all but one of these software sellers are selling very similar products, event stream processing / continuous query engines. There is one industry analyst (a vendor). There is only one independent consultant, evangelist (another vendor).  There are no end users or other industry segments represented on the SC. In a nutshell, the EPTS is composed of vendors, most of which are software vendors selling very similar products.

I propose that the event processing community at-large would be better represented and served if the EPTS SC was reorganized, something like this:

#	Industry	Primary EPTS-Related Technology	Comments
1	Software Company	Event Stream Processing / Continuous Query	Vendor
2	Telecommunications	Network Management	End User
3	Financial Services	Security and Risk Management	End User
4	Software Company	Neural and/or Bayesian Networks	Vendor
5	Software Company	Rules Engine	Vendor
6	Military / Government	Data and Sensor Fusion / CEP	End User
7	Transportation	Operations Research	End User
8	Independent Consultant	CEP Promotion, CEP Web Site Sponsorship, CEP Evangelism	Vendor
9	NPO	Event Processing Standards	End User
10	Open	Open	End User

The above list could certainly be improved and refined. It is just a first draft. However, as readers can see, instead of being a society that is managed by vendors with a single, dominate software technology, the society shifts to one where various industries and technologies are more equally represented. The majority of the SC members in my proposed reorganization would be end users, not vendors.

Your comments and suggestions are warmly appreciated.

Filed under: Complex Event Processing, EPTS, Event Processing, Event Processing Technical Society, Event Stream Processing, Standards | 2 Comments »

It is great to relax and just enjoy the good life in Northern Thailand.  As I read so many blog posts and technology articles by people who just don’t have a clue about what they are talking about, I am reminded of a passage by one of the world’s great writers Umberto Eco.

Foucault’s Pendulum was the second novel written by Umberto Eco. The plot revolves around three friends, who work for a small publishing company in Milan. After reading many manuscripts about occult conspiracy theories they decide they can do better and begin to invent their own conspiracy for fun.  Their conspiracy, which they call “The Plan”, is about an immense and intricate plot to take over the world by a secret order descended from the Knights Templar.  As the game progresses, the three men slowly become obsessed with the details and their game turns dangerous when others learn of The Plan and believe that the three have actually discovered the secret to regaining the lost, vast fortune of the Templars.

In this amazing novel, there is an immortal, Comte de St. Germain, who is central to the story.   During a long ocean voyage, standing desk-side on top of a large vessel watching the sun go down, one of Comte de St. Germain admirers asks, in effect (paraphrasing),

“It must be great to be immortal?” (paraphrasing)

The next few passages in Eco’s narrative are unsurpassed in beauty and wisdom, as Comte de St. Germain laments the painfulness of watching humans born into the world, grow, learn and die, passing along very little knowledge to the next generation, who are doomed to repeat the same mistakes over and over again.  This, in a nutshell, is a metaphor of the human condition.

So my friends, as my hair slowly, but surely, turns gray, and I read so many opinions from those with less experience, I feel pain as did Comte de St. Germain in his amazing immortal journey.  Granted, one lifetime and three decades of technical experience, including zero day Internet distributed computing and network systems engineering experience, is a far cry from immortality; however, I can still feel the pain of reading and watching so many with less experience make so many mistakes, or boast of their new “discoveries”.

It is almost unimaginable to think about how much pain one would feel after living hundreds of generations, like Comte de St. Germain, seeing “experts” with little clue about what they are doing, talking as if they actually knew what they were talking about.

Sigh.

Filed under: Cyberstrategics, Off Topic | 1 Comment »

Paul Vincent posts The Eight Fallacies of Distributed Computing where he and his colleagues state that “essentially everyone” makes these assumptions when designing a distributed computing application.

Essentially everyone, when they first build a distributed application, makes the following eight assumptions. All prove to be false in the long run and all cause big trouble and painful learning experiences.
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn’t change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

I have been designing and building (or architecting) distributed network andInternet applications since around 1988, and I cannot recall one design team who ever made any of the eight assumptions quoted above.

Competent network systems engineers simply do not make these assumptions.

Filed under: Cyberstrategics, EAI ESB & SOA | No Comments »

In August of this year, we issued the challenge,  The Bot Hunter: An Event Processing Challenge (Bot or Not). Not surprisingly, none of the “self-described CEP vendors” with grand claims of CEP greatness, responded.  Real-time detection of threats and opportunities, the core premise of CEP, is very different than algo trading, order routing and process orchestration. Solving real problems is quite different than marketing.

One would think that vendors would take advantage of a real CEP challenge to “show their stuff” versus the releasing marketing fluffy awards.   Folks believe in solutions tested by independent experts, not self-proclaimed proclaimations of greatness.  Every major company has a web site, so it is quite useful to create a CEP solution that can accurately detect automated bots in real-time (with low false positives and false negatives) and then block the bots with high confidence.

One problem is that it is non-trivial to write a set of rules that examine all the real-time web log entries and determine, with high confidence, which transactions are from bots and which are from humans.  This useful application is a bit “complex” and hence, a great CEP application.   Unfortunately,  CEP vendors, claiming “quick, low coding application development” can’t silence their critics!

Readers might recall from The Attack of the Spiders from the Clouds my musings on how cloud computing services can easily be used as a platform for massive denial-of-service attacks.  This is a real-time threat, ripe for a CEP solutions, but we cannot find a single CEP vendor that can meet the challenge.

Filed under: Cloud Computing, Complex Event, Complex Event Processing, Cybersecurity, Event Processing, Systems Engineering, Threats and Vulnerabilities, Use Cases | No Comments »