Abstraction Hierarchies in Event Stream Processing

by

David Luckham and Roy Schulte

This article represents the personal opinion of the authors and does not reflect the official position of Gartner Inc. or any other organization.

Event Stream Processing (ESP) platforms support applications that operate on moving time windows of streaming event data to address a variety of business problems. They can calculate simple aggregates (e.g., count, sum, average, maximum, and minimum) or detect more-complicated patterns in streaming data.

In a previous article we proposed extending ESP platforms to provide complex-event processing (CEP) capabilities to support an explicit and complete notion of causality between events. This in turn would make it easier to deploy more-powerful event pattern description facilities in ESP systems.

In this article we explain how ESP platforms can be used to abstract events according to predefined algorithms and to organize those events into a hierarchy according to their level of abstraction, all happening in real-time. Event hierarchies will provide insights that capture critical business situations at various levels of management within an enterprise on the fly. This will enhance management’s situation awareness and improve decision making in real-time.

Stream Analytics versus Traditional Business Intelligence

Stream analytics is a kind of real-time business intelligence (BI), where, broadly speaking, BI is a discipline that applies mathematics to data to derive insights enabling better business decisions.

Most conventional BI operates against snapshots of data and uses relatively simple math. For example, a BI tool reads an input file that contains detail records of all product sales for the day. It generates a daily report that summarizes the total number of sales (count), average price (average) and total sales revenue (sum) by product. This is an abstraction hierarchy based on simple aggregate calculations. It turns low level raw data (sales line items) into higher level, abstracted information (counts, averages, and sum totals) that helps managers make decisions on inventory management, re-ordering, pricing, promotions, and staff assignments.

Definition: An event is an abstraction of a set of events if it summarizes the information in the set.

An abstraction does not need to contain all of the information in the set. The goal of an abstraction is to make the data contained in a set of events more understandable, processable or easily accessible.

A BI system is likely to report multiple levels of abstraction based on different dimensions in the data. For example, the data may be reported by product or by store. There typically is a hierarchy consisting of several levels of abstraction – for example, sales by department, store, state and country. Each dimension and each level of hierarchy may be used for different kinds of decisions. The information is often plotted graphically to help people recognize relative magnitude and trends. A report may compare current results to projections, past results, results from disparate geographic areas, or data from competitors. It may also call attention to particular results that are unexpected or require attention for other reasons.

ESP platforms are used similarly, but with two differences: (1) they operate in real-time and (2) they can be used to find temporal or spatial patterns in event data.

1. Real-time

An ESP application typically runs continuously throughout the day, monitoring moving time windows of data and constantly updating its results. The input data is processed as it arrives in an unbounded sequence of records (“data in motion”) rather than as a snapshot of data at rest in a database.

This matters when quick action will lead to better business results. The system may display results on a real-time dashboard, send alerts to a mobile device, or trigger an automated response. Alarms and automated responses occur when the real-world events occur rather than waiting for a periodic report (e.g., the daily batch run). The underlying aggregate calculations, the abstracted higher-level events, and the detection of results that require attention may otherwise be the same as in a conventional BI report. An ESP application could report sales by product and product category, or by department, store, state, and country just like a BI report, but show minute by minute results as they occur rather than once daily.

If there is no business benefit to near-real-time sales reports, then the ESP application wouldn’t be any more valuable than a traditional daily BI report. However, for many operational situations, current information does matter. For example, if a contact center manager has a near-real-time ESP report on customer “on-hold” wait times, he or she can intervene immediately to take remedial action such as re-assigning agents from outgoing calls to temporarily handling incoming calls.

In some situations, a conventional (batch) BI report can approach the timeliness of stream analytics by running more frequently, e.g., every few minutes. However, batch reports that are computed from scratch for each iteration require more processing overhead than an equivalent ESP application that computes incrementally as new data arrives. Frequent periodic batches are practical if the volume of data is relatively small and really immediate (low latency) results are not required. However, an ESP application or a custom application with similar real-time streaming logic is required if the business needs information within seconds and the volume of input data is high.

2. Temporal or spatial patterns

ESP platforms are well suited to detect temporal or geospatial patterns in data streams because of their ability to organize complicated and disorganized incoming data – e.g., one or more event streams. ESP platforms have internal frameworks that sort event data as it arrives according to predefined stream processing rules (“queries”).

Most ESP platforms also have purpose-specific event processing languages, some based on SQL extensions, that make it relatively easy for the system designer to express logic that applies to the relative timing of events. For example, some provide built in functions to find when two related events happen within “n” minutes or seconds of each other; when events happen in a certain order (“A” followed by “B”); and/or the absence of events (“C” did not happen in this time window).

ESP applications are commonly targeted at temporal patterns, but sometimes they deal with geolocation patterns related to proximity or lack thereof, or both temporal and geolocation attributes. For example, if a credit card is used in two locations that are 100 miles or more apart within a one-hour time window, one or both transactions may be fraudulent. The combination of the two credit card events (attempted transactions) is abstracted to a higher-level event, called “potential fraud” which is meaningful to a decision maker (a person or automated system). Neither of the lower-level input data (events) indicate a problem by themselves. This kind of abstraction, based on event patterns, is more sophisticated than abstraction based on simple aggregation. Of course, it is possible to write complicated pattern logic in a conventional programming language and apply it to historical data at rest in a database, so an ESP platform is not the only way to implement this kind of abstraction. However, when pattern-based abstraction must be computed in near-real-time on high volumes of data, an ESP platform is a much more effective and efficient kind of tool to use.

A Real-Time Event Abstraction Hierarchy

An event abstraction hierarchy is a hierarchy whose members are events rather than other kinds of data (such as state or reference data). Each level contains events that are abstractions of sets of events at the previous (lower) level. An event that is an abstraction of a set of events is said to be “at a higher level” in a hierarchy of events. In this section, we describe event abstraction hierarchies that are computed in real-time from multiple ESP applications as new events arrive. We illustrate this by considering an e-commerce application (See Figure 1).

Figure 1. E-Commerce Abstraction Hierarchies

Events on the lowest level, base events (sometimes called raw events, primitive events, or just “data”), come from external sources. In our example e-commerce application, they consist of actions that occur on the website and are recorded by software events. Events are typically implemented as messages which may be sent to an ESP platform. Base events in this case include clicking on links, entering a customer name or password, searching for items, selecting items, resizing images, putting items into shopping carts, entering credit card type, entering credit card number, entering credit card security code, and so forth. Every customer causes a stream of events to be generated as they navigate through the website (bottom level in Figure 1).

Pattern-based hierarchies

A stream analytics application, running on an ESP platform, processes base events in real-time as they are received to generate higher-level, abstractions (complex events) (middle level in Figure 1). The figure shows events that are abstractions of sets of customer activities. For example, an “add credit card” complex event on the middle level is an abstraction that combines three base events – enter credit card type, enter credit card number, and enter credit card security code. Similarly, a product lookup event is an abstraction of a sequence of multiple page views as a customer follows links to a particular item, resizes images, and clicks into product descriptions.

The stream analytics application may then abstract sets of mid-level events to indicate that a person looped through the same pages and saw the same and similar products several times without placing any item into a shopping cart. The fact that nothing was placed into a cart is itself a significant event after a certain amount of time has elapsed. This pattern may constitute an unsuccessful search situation, a third level abstract event that is a business issue (top level in Figure 1). This event may trigger an automated intervention, such as a recommendation for alternative products, a pop-up chat bot, or an offer to converse with a human sales agent.

This abstraction hierarchy is based on pattern detection – sequences of different kinds of events happening in a certain order within a certain time interval. The design of abstract events (complex events) is a choice made by the system’s author. A small subset of a customer’s website action events may be represented in a high-level event. Or a customer’s entire web session could be abstracted into one “customer visit” event, depending on the purpose of the stream analytics application.

Simple aggregation-based hierarchies

Another stream analytics application that operates simultaneously on the same, low level, input event streams may only look at completed purchase events and use an entirely different event hierarchy. For example, it may summarize multiple base events into fewer second level abstracted events that contain a count of the number of sales by product in a department. This new application may then abstract the second level departmental sales totals into sales by store (third level), state (fourth level) and country (fifth level) similar to the hierarchy described earlier for traditional BI reports (not shown in Figure 1).

Another stream application, using the same base purchase events but with a different aggregation-based abstraction hierarchy, might track revenue by payment type (cash, different credit card companies, store credit).

Application variety

Pattern based hierarchies often (but not always) reflect a combination of different kinds of events, whereas simple-aggregation-based hierarchies always combine the same kind of data (to compute count, sum, average, max, min etc.). The order of the events and their relative timing does not matter when computing a simple aggregation, although all of the events must occur within the same time window (e.g., all sales within the past hour). An abstraction hierarchy can use both patterns and simple aggregations¹.

One abstraction hierarchy may serve different business purposes for different people. There is often a loose correlation between the level of event abstraction and the level of businessperson using the information: the higher the level of event, the higher the level of person that is likely to use that information. For example, a department manager may need to see detailed sales for each product within their department (level 2); whereas the store manager may care about the sales by department and store (levels 2 and 3); and the regional manager may focus on sales by store and state (levels 3 and 4); and so forth.

Multiple hierarchies may be simultaneously computed from the same incoming event streams to address different issues for different users (see Figure 2).

Figure 2. Different Hierarchies from Same Input Event StreamsOur e-commerce website has many potential event hierarchies including:

A hierarchy for operational monitoring, implemented by an Application Performance Management (APM) tool and used by application teams to monitor the throughput, latency, availability and resource consumption of the website.
Other infrastructure and operations hierarchies, implemented by system and network managements to monitor the workload, overhead and reliability of the equipment.
A hierarchy of customer web experience statistics that report total customer views and number of customers active on the website at one time.
A hierarchy for detecting instances of possible customer fraud.
A hierarchy for detecting abandoned-shopping-cart events to automatically trigger a process to reach out to the customer via email or message with a follow-up offer.

Conclusions

Base events are input data, abstractions are computed.
Abstractions may be based on temporal or other patterns; simple aggregation; or a combination of those.
Abstractions that are simple aggregations are based on homogeneous data, usually involving events of one lower-level event type. Aggregations that are based on complicated patterns are usually based on a set of heterogeneous event types.
The number of higher-level events is almost always far smaller than the number of lower-level events.
Every stream analytics application has at least two levels of abstraction hierarchy – a base event level and a complex event level. It may have three or more levels but will rarely have more than 6 or 7 levels.
Many kinds of abstractions can be computed from the same input base events.
An organization will need different higher-level events in different abstraction hierarchies according to the interests of different departments and decision makers.
Event hierarchies are just one of many kinds of information abstraction hierarchies. Some of the principles of event hierarchies described here also apply to non-event kinds of hierarchies².

Computing event hierarchies in real-time as data arrives is valuable because it enables organizations to act quickly on threats and opportunities as they emerge. However, event abstraction hierarchies can also be computed in a nightly batch run or in an ad hoc, interactive application that operates against event stream data that was collected and stored weeks, months or years in the past (so it is far from real-time). In fact, post hoc analysis of events (i.e., analytics on historical event stream data) is fairly common for forensic purposes. It may be used to determine what went wrong (or right) in the past.

Stream analytics and the principles of abstraction hierarchies are also applied in simulations. Much of the theory and practice of using abstraction hierarchies on event data was developed at Stanford University for the Rapide system in the 1990s and originally used for non-real-time discrete event simulation.

Footnotes

Terminology: Some people consider a simple aggregation like count or sum to be a kind of “pattern” of homogeneous data items. Others consider patterns composed of heterogeneous event types to be a kind of complex “aggregation” because they combine multiple lower-level events.
“Non-event” hierarchies: Hierarchical structures appear in many aspects of information processing and elsewhere. For example, an inventory report of products sitting in a warehouse is a data hierarchy with multiple levels of aggregation, but the data is state data not event data. An organization chart for a company is a different kind of hierarchy, also not based on aggregate calculations or patterns of lower level data.

Abstraction Hierarchies in Event Stream Processing