Continuous Business Intelligence and Big Data

by: Philip Howard, Research Director, Bloor Research

I have written previously about how the major vendors of CEP (complex event processing) solutions are still primarily focused on either capital markets or supporting business process and SOA and are not really focusing on business intelligence and analytics per se. The exceptions are IBM and StreamBase. However, this is by no means true when it comes to the operational and big data markets. In the former area, Tier-3 and Red Lambda both use CEP for SIEM (security information and event management) while AccelOps employs it for data centre monitoring.

However, it is on big data more generally that I want to focus. Here there are a number of new and not so new ventures that provide what is becoming known as “continuous BI”. First, it is worth explaining what continuous BI is. Normally, when you run a query that’s it: it’s a one-time shot. If you want to run the same query again then you have to activate it again. A continuous query, on the other hand, is one that you fire up and, once instigated, it continues to run until you tell it not to. What happens is that the results of the query change as new data passes through the query. It is, if you like, a sort of sophisticated real-time monitoring that doesn’t simply record what has happened but supports complex analysis and pattern matching (and, of course, simpler things) against the incoming data.

Now, this is typically a big data problem but it won’t work to simply load the data into Hadoop and then query it. That won’t be continuous. So you need some sort of CEP engine to do it for you. Four such products are (or will be, they are not all commercially available yet) S4 from Yahoo!, which is part of the Apache Incubator programme; Storm from BackType Technology, which describes its product as “The Hadoop of Real-time Processing”; DarkStar from Cloud Event Processing, which provides CEP-based streaming MapReduce; and HStreaming (from the company of the same name). As its company name implies, DarkStar is a cloud-based and HStreaming also offers a cloud-based version. ……  Article