Processing Complex Events at the Large Hadron Collider (LHC)
Here is an excellent Google TechTalk by Dr. Majorie Shapiro (LBL & LHC ATLAS), June 18, 2007, on processing complex events at the LHC called, Supersymmetry, Extra Dimensions and the Origin of Mass: Exploring the Nature of the Universe Using PetaScale Data Analysis.
This is really what I would call “real-time CEP.” Bean collisions at the LHC happen every 25ns, and all the events produced are pipelined for decision processing to distinguish an interesting event from a non-interesting event.
The detection process uses a similar concept we outlined in our CEP functional model (based on levels in the JDL functional data-fusion model). Level 1 triggering preprocesses detected events with specialized detector hardware. Preselected Level 1 events are processed by computers looking for regions of interest (Level 2). These results are further correlated (Level 3) with results from the entire detector array (the whole complex event) and then sent to tape for archiving and future off-line processing.
Instead of explaining ATLAS at LHC (which I am not qualified to do), I recommend you watch the video yourself; it runs about one hour and 15 minutes. The video covers the computing processing required to process these complex events. Query processing, data storage requirments, raw bit stream processing, hierarchical event processing, event summary data, event tagging (classification for query purposes). event data statistics (amount, size), event generation simulation, distributed analysis, data mining and more event processing topics are covered.
Best quote in the talk:
“… how you do the analysis? - It ain’t SQL queries !! ” - Dr. Shapiro (0:48:44)
david luckham says:
Thursday, June 11, 2009 at 9:05am
Tim,
would you be willing to expand this blog post a bit as an article for the website?
I think more detail on levels 1, 2 ,3 would be appropriate for those who don’t have time for the full video.
Say 1,000 words?
It would still reference the video for “full details”
cheers
- David
Tim Bass says:
Thursday, June 11, 2009 at 2:15pm
Hi David,
Nice to hear from you. Thanks for visiting.
Yes, I am happy to do a more detailed write up in the future (I will post here and you can repost as you like!).
Having just returned from vacation, I am behind in a number of tasks, including a write-up on performance test results with the Amazon S3 / CloudFront team, a write-up on CloudBerry (used to help manage S3) and a number of other projects (I am currently and painstakingly reading a science text on how the nerves in the foot relate to the health of the organs, and the book is in Thai (!), not English, so I this is very slow reading!!! and I am doing some novice physics research on Kaluza-Klein dimensions as it relates to String Theory and braneworld dualism – another project on my desk.)
So, I’ll definately look into a more detailed write-up on ATLAS event detection in the LHC and how processing those complex events relate to the functional reference mode I have offered for event / situation detection for CEP, in the future.
Cheers. Hope you and “the family” are doing well! Take good care.
Yours, Tim
Rainer v. Ammon says:
Tuesday, June 16, 2009 at 4:14pm
Hi Tim-san,
good hint! I wanted to answer in this blog directly but then I had no idea how to format a longer text. So, please look at http://www.citt-online.com/index.php?id=veranstaltungen&id3=use_cases&id4=more
I’m still thinking whether I will keep it, change it, delete it…
Best regards
Rainer
Tim Bass says:
Tuesday, June 16, 2009 at 4:35pm
Hi Rainer,
Great to hear from you. Hope all is well.
As I see it, folks have been processing complex events for decades. The technologists that are not processing complex events are the folks using the term “complex event processing”.
The term “complex” is fairly well defined in science and systems theory. It is best to use established definitions and not reinvent decades of research.
In other words, as Peter Lin has commented, most of insular group of people discussing “complex event processing” are decades behind the state of the art. They are so far behind, that there is little to be gained from reading their blogs or books.
David Luckham’s original work on CEP was based on solving problems known to be “complex”, and it is funny, at least to me, to see him using the term “simple complex event processing” to describe the software community.
It is simple event processing, not simple-complex event processing, LOL.
Look at how much spin folks have to spin to keep up the same folly! Let’s stop this nonsense and get back to science and engineering, not marketing and snake-oil software.
Edit: Sorry to be so harsh, but from my perspective, the past few years in CEP-Land have simply been backwards steps in science and engineering. I hope we can start to move forward soon! To do that, the community need to change leadership and expand into established “CEP” domains, like what folks are doing at CERN and many other places around our world.
Yours, Tim