What Is Real-Time Data Streaming—and Why You Should Care

by Mary Catherine O’Connor,  IOT Journal

No company deploys Internet of Things technology because it just wants to use more sensors. It’s the data those sensors capture that matters. Yet, just capturing data is not enough. In order to avoid a business loss—a manufacturing process coming offline, for instance, or a turbine at a power plant failing—sensory data needs to be analyzed in real time, so workers can respond to anomalies that are the harbingers for these types of losses before such losses occur. Simply collecting and storing sensory data, in industrial applications such as these, does not provide any real business value.

“A decade ago, customers with big data would use a data-warehousing approach and look to get reuse from dense stores of data—looking at customer transaction data, for example,” says Chad Meley, VP for unified data architecture product and services marketing at Teradata, a provider of data-analytics services.

Over time, those applications evolved into what are known as complex event-processing (CEP) platforms. Then, as the volume of data grew, the enterprises realized they wanted to analyze it as they were collecting it, resulting in the emergence of tools for advanced streaming-data analytics. This type of software performs three important functions: analyzing a wide variety of event- and time-based data types (a system could consume data from a wide range of sensors, detecting anything from temperature to vibration to light), analyzing it as soon after collection as possible, and comparing the data with models that convey what it should be showing in order to then detect anomalies.

“Typically, the events, such as sensor readings, are ganged together and aggregated in a window based on time or on the number of events,” explains Fern Halper, advanced analytics research director at market research firm TDWI. Filters are used to find only the relevant data from the stream and pull that information into these windows that are only open for a certain amount of time. And as the data is compiled, the software runs calculations. “In complex event processing, you’d do things like take subtotals or averages of data coming through. You might want to track temperature—say, so once every 10 seconds, the software captures a window of temperature data and calculates the average. And then, the next window comes through. And with each window, the software decides if an alarm needs to be sounded, based on temperature thresholds that the user has set.”…….

Adoption of advanced streaming-data analytics is slow, most respondents to a survey indicating it is in their 3-year plan.  …..

All of the major enterprise software providers offer streaming data-analytics software or services: IBM (IBM Streaming Analytics), SAS (Event Stream Processing), SAP (HANA Smart Data Streaming), Oracle (Stream Explorer), Microsoft (Azure Stream Analytics) and Tibco (StreamBase). These programs are available through various models, in some cases as a managed service, through a software license, or as software-as-a-service or platform-as-a-service, and increasingly these are cloud-based offerings. Google and Amazon Web Services also offer cloud-based streaming data-analytics services (Cloud Data Flow and Kinesis Analytics, respectively).

The list of potential uses for streaming-data analytics is nearly as long and varied as the list for IoT technology applications.

“Say you’re tracking an oil rig,” Halper says. “You’ll be collecting all kinds of sensor data [from various systems on the rig].” When it comes to responding to oil-rig failures, she adds, timing is vital. With streaming-data analytics, an oil company can create a model, based on historical data showing what types of output the sensors should be generating, per the rig’s normal operation. “You take that model and put it into event stream,” Halpern explains, and the company will then receive alerts as soon as any of the sensor data exceeds a set parameter….

Read the article in IOT Journal

DCL: Interestingly everyone is now listing “complex event processing” in their inventory of tools for real time streaming analytics. But it usually turns out that only the simplest and easiest to implement aspects of CEP are in use. I expect that as this field of applications develops some will re-invent the more sophisticated aspects, such as event hierarchies and event pattern languages for themselves. Meanwhile, read my  book, “Event Processing for Business” where you will find all of the current use-cases described, and more besides.