Reasoning and Complexity
In a short follow-up to CEP Software Saves the Universe!, I thought it would be helpful to post some quotes from Wikipedia on reasoning and complexity.
Cognitive science sees reasoning by the analogy to a data processing, where relations between observed properties of reasoning are used in numerous models leading to evident logically correct conclusions in different circumstances.The complexity and efficacy of reasoning is considered the critical indicator of cognitive intelligence. Therefore it is the inevitable component of cognitive decision-making.
In general usage, complexity tends to be used to characterize something with many parts in intricate arrangement. In science there are at this time a number of approaches to characterizing complexity, many of which are reflected in this article. Seth Lloyd of M.I.T. writes that he once gave a presentation which set out 32 definitions of complexity.[1]
Definitions are often tied to the concept of a ‘system’ – a set of parts or elements which have relationships among them differentiated from relationships with other elements outside the relational regime. Many definitions tend to postulate or assume that complexity expresses a condition of numerous elements in a system and numerous forms of relationships among the elements. At the same time, what is complex and what is simple is relative and changes with time.
Some definitions key on the question of the probability of encountering a given condition of a system once characteristics of the system are specified. Warren Weaver has posited that the complexity of a particular system is the degree of difficulty in predicting the properties of the system if the properties of the system’s parts are given. In Weaver’s view, complexity comes in two forms: disorganized complexity, and organized complexity. [2] Weaver’s paper has influenced contemporary thinking about complexity. [3]
If you are in the world of capital markets and are a big fan of “feeds and speeds” (frankly, I find “feeds and speeds” boring event processing, BEP), please keep in mind that complexity is not defined in science and engineering based on speed (or latency). For example, you might have a complex system that has many components and intricate relationships among the elements, however, the speed at which the component communicate, move, or are processed are orthogonal to complexity.
For example, you might have a very simply Pinewood Derby car and a complex Lotus. The Lotus is complex, not because it can go so fast, it is complex because it has many intricate parts and so many relationships among the parts. If you take the simple wooden car and the complex Lotus and launch them in space, where they travel side-by-side at the same speed, the Lotus remains a complex engineering marvel and the pinewood car remains a simple wooden structure.
Generally, when thinking about “complex event processing” we should think about processing complex events, which are events with numerous intricate components and myriad relationships between the components. In capital markets, an example of CEP would be to process “the cloud” of market data and determine, for example, the relationships between various components (causality). We often hear analysts say “The market moved up today because of news from blah blah blah.” However, most of the time the analysts opinions are mostly “blah blah blah”, because they really have little clue about what is really moving the markets. (If they did, they would be very rich traders.) If systems were processing complex events, they would be able to ascertain, with high accuracy and confidence, what is really causing the markets to move.
There is a lot of chatter in the “CEP network” from folks who say “Oh, forget about this useless debate about complexity, who cares!!” The truth of the matter is that many people care because they are interesting in solving complex problems, not simply processing a stream of event-objects and calculating a VWAP or routing a market order. Orchestration, scheduling, and filtering (here, VWAP is a kind of filter) are, generally speaking, simple event processing.
These distinctions are very important to the advancement of the state-of-the-art of complex event processing. Those who want to censor the discussion on reasoning, complexity and how to processing complex events and situations are self-serving and not serving the large community with complex problems to solve. Simply routing orders better and faster, or calcuating a VWAP faster, is great; but this is not complex event processing.
