Complex Event Processing in the Belly of the Beast
Around 10 years ago I worked as Technical Director for SAIC as a member of an elite group of Internet security experts. I was responsible for a number of very large global financial corporations (well, actually two). The leader of our division, called The Center of Information Protection (The CIP), was a very intelligent lady and she used to refer to the corporate economic engine (in general) as “the beast”. She often reminded me that, as members of a large corporate entity (SAIC), we had to “feed the beast”. We used to laugh about this, as if “the corporate beast” had the numbers 666 stamped on the front door in the lobby.
Fast forward to recent history.
I read the original papers on complex event processing by David Luckham and I was excited to read David discussing the same technologies I was working on with clients like the USAF and DOD; where we were trying to solve numerous “needle-in-they-haystack” detection problems. CEP seemed promising and it had it roots in network management, security management, network (cause-and-effect) analysis, sense-and-respond networking (like the Boyd OODA loop) and other complex areas where we have myriad yet unsolved critical problems.
However over time, this good concept of making progress in solving needle-in-the-haystack problems in cyberspace somehow became hijacked by people in algorithmic trading corporate America. Since most “CEP Engines” could not (and still cannot) actually solve complex needle-in-the-haystack problems, software companies quickly found a niche market with folks in financial services who wanted to build quicker algorithmic trading applications.
Frankly speaking, renaming algo trading platforms as “CEP engines” or “event processing platforms” has been a great disappointment for me and has done nothing to advance the state-of-the-art of solving complex problems in cyberspace. CEP has been, for far too long, consumed in the belly of the beast.
I continue to read a number of news articles critical of algorithmic trading, recently, for example, SEC Seeks To Ban High-Frequency Trading by Jenny Anderson. I have long held the opinion that corporations that use the stock market as a corporate playground to make money off perturbations in the market, using high speed computing and algo trading techniques, both damage and destabilize our market(s).
I agree with many of the articles I have read in the past few years, some by very experienced people in capital markets, who say that the concept of high-frequency computer trading is completely the opposite of the fundamental market concepts of stock ownership. Stocks should be bought and sold because investors believe (or do not believe) in the company, the company management, and the mid to long-term market potential of the company.
Treating the stock market as some high technology computer game where a corporation looks for hidden perturbations and opportunities, only realizable by split-millisecond sub-market trading is destabilizing and worrisome. This type of computer gamesmanship with our capital markets benefits no one except financial services firms and their profit margins.
David Luckham notes on his recent blog post:
While there can be little doubt that the SEC’s proposals are well justified, and late incoming, are we seeing here the beginning of regulation of the uses of computers and event processing?
I think David is asking the wrong question. The question that David should be asking, in my opinion, is:
Should we continue to associate the original noble concept of concept event processing (CEP) with algorithmic trading, which has nothing do with with our original good work in event processing and is known to be controversial, destabilizing, against investment fundamentals, and problematic?
It is my opinion that the entire technology space of complex event processing would be better off if we (including David) would correctly state, algorithmic trading is not in the domain of complex event processing, it is simply in it’s own domain called, algorithmic (or program) trading. However, it seems the train has left the station and we are carrying the dirty corporate greed garbage with it.
I learned a long time ago not to attempt to second guess the motives of others or to try to read their minds as to why they do things I find wrong, incorrect, unethical or immoral. So, all I can say is that I find the association of algorithmic trading with CEP troublesome and demoralizing. In fact, I find it so troublesome that much of my earlier excitement and support of CEP technologies has diminished because almost all I read in the trade-rags on CEP are press releases by the beast masters talking about how great CEP is for program trading in capital markets.
CEP has, without a doubt, been consumed into the Belly of the Beast and, perhaps, forever tainted by the greedy, selfish corporate behavior that only benefits a few in the financial services sector. CEP used to be fun, and we could talk about all types of interesting detection-oriented problems that need to be solved to make cyberspace a safer, better place for everyone. Instead, the dialog has been hijacked by greedy financial institutions and the software companies, technologists, reporters and analysts that make money in that tainted ecosystem.
I prefer a simpler “Zen Life” of ethics, social responsibility and integrity, and a few beers by a lazy river, than the seduction of heaps of money, huge bonuses, and an overly consumer-driven lifestyle where nothing matters but gaining more money, power and material possessions.
Regarding CEP, the cat is out-of-the-bag. Pandora’s Box is wide open. There is no turning back, it seems.
In closing, if I was the SEC, I would ban most forms of high-frequency computer trading and split-second market manipulation and return our capital markets back to the hands of mid-to-long-term investor-owners. I sincerely hope the SEC is successful in regulating the beast masters in financial services who have done much to damage our way of life and destabilize our nation and our world.
David, it is not about regulating computers that process events, my friend. It is about regulating the beast and the beast masters who ride the back of the big green monster; and about returning our stock markets back to the fundamentals of company ownership via investing in stable markets that are not dominated by beast masters playing games with machines (and our future).
Filed under: Agents, Analytics, CEP News and Events, Complex Event Processing, Cyber-Trading Technologies, Cybersecurity, Event Processing, Financial Services, Threats and Vulnerabilities












For what it’s worth. We are at least trying to solve problems which feel ethically more sound. We are for example currently looking for participants for a research project where we try to add interesting capabilities to our CEP engine so it can hopefully give early-warnings when transports with hazardous materials are doing dangerous things. Oil tankers come to mind and I would be happy to just increase the probability by a few percent that we can detect interesting behavior so that we can avoid accidents with an oil tanker. Smaller examples are all tankers on our roads with chemicals.
Good post. Having worked on trading systems and see how some hedge funds and mutual funds operate, I couldn’t agree more. Many firms buy pre-trade/post-trade compliance software to get the SEC off their backs, but don’t run it. Literally, several of the clients I talked to said “we don’t run it.” I got different answers as to why they didn’t run it.
The real reason is they don’t want to. Many firms rather pay fines every quarter than obey the law. Many firms pay millions and tens of millions of dollars worth of fines every quarter.
Like you, I find the current trend disturbing and hope the people and SEC turn it around.
peter
I loved the movie about Sirius XM: “Stock Shock” because it explains how the whole naked short selling stock market manipulation thing works-and how the company nearly went bankrupt. Good DVD. Amazon has it or stockshockmovie.com has a movie trailer.
Tim,
Let’s make CEP fun again - how about some use cases that the current products can’t address all that easily (and no algo trading please!).
Colin
Tim,
I agree with much of what you have to say here. More at http://colinclarkeventprocessing.com/?p=296