Complex Event Processing and The Future Of Business Decisions

1. Introduction

Complex Event Processing (CEP) is becoming increasingly important in business decision making. CEP is a basic component of a type of real-time business analytics used in continuous intelligence. Information contained in business events is extracted and analyzed to provide insight into the changing conditions in a company and its environment. A CEP application typically operates on multiple incoming observed events, sometimes only a few but sometimes thousands of events per second, to derive a much smaller number of more-significant, complex events that summarize the data in those lower level events. It enables business decisions to be made and actions taken within milliseconds, seconds or minutes of receiving new event data regarding current conditions. Thus CEP is used to improve situation (or ‘situational’) awareness, which is defined as “knowing what is going on so you can decide what to do”.

CEP is a new kind of event processing that utilizes old and new ideas about events. Of course event processing has been a fundamental part of computer systems for decades. But CEP targets the use of event streams at very high levels of business management and decision making. New ideas in event pattern detection, event abstraction and hierarchies of events are involved.

CEP is becoming more popular, thanks to advances in event-processing software, the growing understanding of CEP design issues, and the need for faster and smarter business processes. When people refer to “event processing” today, they often are thinking of CEP rather than the simpler and older aspects of event-driven systems.

The goal of this article is to examine the future of CEP and its role in making decisions. We begin with some examples of current CEP business applications in Section 2, and give a precise definition of CEP in Section 3. The future is discussed in section 4.  

2. Four Current Business Applications

2.1 Retail Banking

Two of the biggest challenges in retail banking today are fraud detection and cross-selling. Banks are assaulted by people trying to find vulnerabilities in the banks’ applications so that they can overdraw their accounts or gain access to other people’s accounts. Part of the problem is that checking, saving and credit card systems are stovepipes, so each may be unaware of events that have occurred in the other systems. Moreover, there may be a temporary lag in transferring information about transactions that have occurred in dispersed locations, such as branch offices, automated teller machines and on the Web. Crooks are sometimes able to exploit these gaps in knowledge transfer to trick the systems in allowing excess or unauthorised withdrawals.

A leading-edge bank is using a CEP-based application to continuously monitor more than 50 types of events. It correlates information about the transactions that are occurring in multiple locations and systems, typically within a second or two after they happen. The CEP application detects patterns of transaction events that indicate a likelihood of fraud, and stops the processing or promotes the issue to a person for human adjudication and follow-up. Examples of such event patterns are a customer opening multiple new accounts at several branch offices within a few miles of each other, which may indicate that the customer is setting the stage for fraud. Or a series of credit card charges, starting at a few cents and then escalating the amounts, thus testing the credit card processing system’s ability to detect unauthorized access.

Interestingly, many of the events that are relevant to detecting fraud are also relevant to near-real-time cross selling to legitimate customers. Another CEP-based application in the same bank continuously monitors large deposits and withdrawals, and address and name changes, to find indications of life events such as marriage, moving, buying a house, or the arrival of a baby. This current event data, combined with reference data about the customer’s demographics, guides cross-selling recommendations used by tellers, platform bankers, call centre service representatives and automated Web banking systems.

2.2 Long haul truck fleet management.

A national trucking fleet has satellite navigation systems and wireless-enabled controllers installed in all of its long haul trucks. The objective is to reduce the costs of operations by improving the routing of trucks, optimizing the assignments of cargo orders to trucks and the scheduling of drivers.

A truck’s on board system sends events by wireless to regional fleet control stations. Each event contains data about the position and condition of the truck. The system communicates with the nearest regional control center governing the truck’s current position. This continuous feed of events from each truck includes a lot more than just its position. The on-board system monitors fuel level, temperatures in the engine and cargo compartments, vehicle speed, tire pressures and other performance parameters. It also monitors trip data such as the expected time to the next scheduled route stop, and the truck’s progress with its delivery and pickup schedules.

A regional control center monitors the event streams from each truck in its region. Each truck has a trip plan that has an associated set of constraints for that trip. The constraints are based on the type of truck, the cargo, load factors, delivery and pickup schedule, driver’s work schedule and many other parameters. The event stream from a truck is monitored for conformance to its constraints. A control center also factors historic data on routes, traffic conditions and current weather reports into its decisions.

As a result of these event inputs, the event processing system at a regional center may trigger various alerts and instructions that are communicated back to the truck and driver. The intent is to keep a truck on an efficient trip plan, conform to driver safety policies and to minimise costs such as fuel consumption even when adjustments to the original plan are required.

The automated event-driven tracking of trucks also can help to optimize trip times.  For example if a tanker truck is expected to pick up a load at a port which requires a special equipment setup, the regional center will give the port an arrival alert 15 minutes in advance. The alert allows the port crew to be ready to load the tanker instead of it idling while equipment setup takes place.

A regional control center communicates events summarizing the current states of the trip plans of trucks in its region to a national control center. Events received at the national centre are therefore aggregations of events at the regional centers. A hierarchy of status and control events for the whole trucking fleet in motion is thereby computed in real time. The national control center has a view of the trucking fleet operation in progress across the entire country. It may communicate instructions back to the regional centers on the management of the trucks in their region. These instructions might include reassignments of cargo pickups to different trucks, changes in truck routes, driver assignments, and so on. Planning of future assignments, maintenance, schedules and long term fleet planning is done at the national center.

2.3 Monitoring Customer Service Processes

One of the leading uses of CEP is to improve the monitoring of business processes involved in customer service. For example, successful operation of an electrical utility depends on effective processes for bringing new customers on board, and for temporarily disconnecting and then reconnecting customers who have payment issues. These processes are driven by events in four major application systems: a CRM application for entering new requests and other aspects of customer interaction; a billing application; an EDI application for sending transactions to external partner service providers that do the wires, poles and physical connections; and a transformation application to prepare data to be sent or received through the EDI network.

Prior to the introduction of a new CEP-based monitoring system, each of the applications had its own “stove-piped” management capabilities, none of which had visibility into the other applications, so they could not provide an end-to-end view of the processes. Therefore, some customer requests got misplaced and languished untouched in the seams between the applications when communication between these processes went wrong or handoff events between the steps in a process were lost.

A large electricity utility solved this problem by implementing a system to monitor the customer support processes from end to end. The system tracks every new customer registration or disconnect/reconnect transaction through 9 milestones in processes that last for several hours or up to several days. The monitor has a model for how each transaction (process instance) is supposed to progress, and how long each step should take. It detects an event whenever an application completes a major task, and the system correlates this event data according to the unique transaction identifiers. If any step is not completed within a reasonable time frame, the system realizes that a transaction is stuck, alerts a manager, and guides them through a problem resolution process. The system also monitors the health of the overall utility operation so that staff can see if the applications and networks are running properly, and the human management activities are being completed. The result is faster problem detection and correction, better service levels, and less customer churn.

2.4 Event processing in health care management

Healthcare is a growth area for event processing applications. Computers and information systems have been used for collecting patient data in healthcare for 50 years. But progress towards a unified national healthcare delivery system has been slow. Currently many different specialized systems are employed, each with part of a patient’s treatment record, and no central system with a complete history. As a result, a lot of relevant information is never included in evaluating a patient or a treatment plan. And costs of medical care have not been reduced as much as desired by these systems. This situation obviously has to change.

A vision of the future is one national medical information system across all hospitals, all patients, and all diseases, conditions and diagnostic tests. One goal of the system would be to automate the coordination and delivery of a patient’s treatment over his or her entire lifetime history. Another goal is to enable remote medical diagnosis and treatment by consultants and specialists who may be far away from a patient’s location. Everything must be done in right now time. And all relevant information must be immediately available to those medical staff who need it. These systems will be event-driven.

2.5 Event-driven processes in hospital operations*

Types of events input to a care management system will vary from the events output by medical equipment monitoring a patient’s vital life signs, the results of medical tests from laboratories, and status changes of medical equipment, to the disposition of operating theatres and other critical hospital resources, the location and disposition of medical staff on duty, and so on. Some examples are:

• Equipment status events such as, E in service busy, E in service available, E in maintenance, … where E might be an equipment or operating theatre.
• Events such as acts of accessing medical IT systems, e.g., EMR (electronic medical record) access by X, RIS (Radiology Information System) access by X, Update to EMR, Update to RIS, … where X is a person with access privileges.
• Medical tests and their status changes, e.g., blood test sent to lab., blood test results ready, patient record updated by test result, radiology report ready, examination report entered in EMR,
• Events signifying changes in the status of doctors such as on duty, is available, is unavailable, is assigned to task T, etc., where tasks (e.g., in conference, in operating room, in diagnosis, updating patient record) can vary in urgency and affect the doctor’s availability (e.g., interruptible, not interruptible, …). Similarly for other types of medical staff.
• Events tracking the status of patients, such as P entered into ER system, P assigned exam room, P under evaluation, P in radiology, P in OR (operating room), …

What we have described is an event processing management system built up from smaller specialized systems. The types of events output by the system include events responding to requests, e.g., for patient records, test and radiology results, and also events resulting from monitoring the hospital processes such as events to keep those processes on track, e.g., patient treatment alerts, equipment and staff assignments, equipment maintenance orders, operating room cleaning orders, etc.

The long term goal of developing this kind of system is to improve the deployment of medical staff and equipment, reduce errors, say in prescribing drugs in treatment plans or reduce duplication of tests, and ultimately help to reduce the costs of healthcare.

* This example is based upon private correspondence with Leendert W. M. Wienhofen and Andreas D. Landmark of the Norwegian University of Science and technology. 

3. What is CEP

Let us review exactly what we mean by CEP.

Complex event processing refers to the processing of representations of events by computer. An event is simply “something that happens” in real life. But to process events, maybe thousands of events, we must represent them in a form suitable for automated processing. Event objects as these representations are called include data such as where and when the event happened, how long it took, if it was caused by other events, etc. Examples of event objects are:

• a stock trade message (an object) that reports a stock trade (an event),
• a weather report input to a weather simulator for weather forecasting,
• a purchase confirmation that records a purchase,
• a signal resulting from a computer mouse click,
• an RFID sensor reading (i.e., an event object that is a set of electrical signals)

More complex examples of event objects are used by computer systems such as simulations to represent or record events (activities) that happen or could happen. Often the purpose of this kind of event processing is to predict what might happen in the real world, e.g., predicting elections or forecasting the weather. These event objects can be any kind of data structure from binary strings to records and other complex data types.

CEP consists of a set of concepts and principles for processing event objects and methods of implementing those concepts. So it is two things, event processing concepts and implementations of them.

Some CEP concepts are well known from other kinds of software systems. Other concepts are only just beginning to enter the state of general practice. For example, one of the key concepts deals with how to specify patterns of events and the elements of computer languages for defining event patterns – i.e., the expressive power of pattern languages. Another is how to build efficient event pattern detection engines.

A third area of CEP deals with strategies for using event patterns in business event processing. For example, it encompasses building systems of event-pattern-triggered business processes and monitoring their execution and performance – both when they run correctly and when errors occur. Finally there are concepts that have not yet entered commercial applications, such as event hierarchies. This area of CEP deals with how to define and use hierarchical abstraction in processing multiple levels of events for different applications within the business enterprise.

The primary focus of CEP today is on the types of higher level events that occur in business information systems and are used in managing the business. That is where the money is. But we note that CEP applies to any types of events occurring in any kind of system.

4. The future of CEP

4.1 Market Definition

Software developers can get CEP functionality by coding it from scratch as part of their application, or by acquiring a general purpose event-processing platform and tailoring it to their specific business requirements. Before about 2004, almost all CEP was custom coded into the application because general-purpose commercial CEP platforms were not widely available. Gartner is now tracking 17 such products, and developers are using them in a growing number of applications. However, companies still get most of their CEP capability by buying a packaged solution with custom-coded CEP logic. For example, security information and event management (SIEM), supply chain visibility, fraud detection, network and system management (NSM), and some financial services trading platform products have purpose-built CEP logic built into the respective applications. In many cases, buyers don’t even know that the product is using CEP under the covers because they deal only with features that are specific to the application.

User companies buy general-purpose event-processing platforms when no off-the-shelf CEP-enabled applications that meet their needs are available. They also buy general-purpose CEP platforms when they have multiple business problems that will require CEP, and they want to use one tool to handle all of their needs. Furthermore, some packaged application vendors and system integrators also embed general-purpose platforms in their solutions when they want to save development time and effort. Commercial platforms are programmed using high-level, event-processing languages (EPLs) that make application development, maintenance and the inevitable ongoing modifications faster and easier.

About one third of the use of general-purpose event-processing platforms is in financial trading, including equity or foreign currency trading. Another ten percent of the use is in smart electric grid projects. Transportation operations, including airlines, trucking, ocean shipping and railroads, account for about another ten percent of the usage. The remaining usage is spread across a wide variety of industries and applications including oil and gas, retail banking, national  security, other governmental applications, healthcare, insurance and others.

4.2 Prospects for Market Growth

Companies, governments and other enterprises are greatly increasing their use of virtually all kinds of analytics because the cost of sensors and adapters to acquire data, networks to move data, and computers to process data continues to drop dramatically. Real-time analytics, including, but not limited to CEP, are growing even faster than other kinds of analytics because it is starting from a smaller base (non-real-time analytics were already widely deployed). Moreover, the accelerating pace of business in general is increasing the emphasis on real-time analytics. Speed is a major component in most modern business strategies, including time-based competition, just-in-time inventory, “zero latency” enterprise, real-time enterprise and other strategies.

CEP is overkill for the majority of real-time analytic applications because the job can be handled by simpler or slower tools, including business activity monitoring (BAM) platforms, visual data discovery tools, business intelligence (BI) reporting tools, log file analysis tools, spreadsheets or mashup platforms. However, some problems can only be handled by CEP. CEP is used when

• the volume of input events per second is high (typically hundreds or a few thousand events per second, but into millions per second in extreme CEP applications),
• the latency of the reaction must be low (typically under 100 milliseconds, but in some cases less than one millisecond between the time that an event arrives and it is processed, resulting in some trigger for action or alert), or
• the event patterns are complex (typical CEP patterns are based on temporal relationships or spatial relationships, but other kinds of patterns can also be matched).

The largest enterprises are deluged by 100,000 to 100 million business events per second, originating in their application systems, sensors, social applications, the Web and other sources. A single financial market data provider, Exegy’s Ticker Plant driving the MarketDataPeaks website, hit 6.65 million messages per second on 7 October 2011. Companies also get countless more low-level “technical” events from IT and network equipment and other physical devices. If the data is dispersed across many different applications in many different locations, CEP may not be required. But if some of it is concentrated, then CEP, either in custom code or executed on a commercial event-processing platform, is the answer. The biggest single source of future demand for CEP may be the emerging “Internet of Things.” Kevin Ashton introduced the term “Internet of Things” in 1999 to describe the concept of assigning a unique identifier to physical objects, and deploying instruments to record the location (and sometimes other characteristics) of the objects. Any person, car, box of cereal, book, or pharmaceutical product can be continuously monitored wherever sensors are available. RFID readers, bar code scanners, and other devices detect the presence of objects and send events through Internet-based event-processing networks (EPNs) to CEP-enabled servers that maintain virtual representations (state data, or in some cases, intelligent avatars) for each object. This makes it possible to track, trace and inventory all of the tagged objects, reducing stock outs (when a company runs out of inventory), lost articles, and theft.

Social computing may be the second largest source of new data and demand for CEP. Every friend request, update to a user’s profile, “like” tag, message posted on a wall, Twitter tweet or other activity on a social site is an event. CEP is used to correlate events that occur in social activity (event) streams to identify things that happen to individual people or particular groups of people. The results help people stay in touch with each other, and can also help companies provide better customer service and more-effective cross selling.

CEP is already being used in some places for the Internet of Things and social computing, but largely within stove-piped, limited domains. Individual companies capture a few specific types of events for a few particular purposes, such as social networking, supply-chain management, cross-selling, or fraud detection. Over time, however, companies will discover the benefits of sharing event data across the stove-pipes so that facts from different origins can be correlated and deeper insights can be achieved. The fraud detection and cross-selling applications in the retail bank, described above, are limited examples of this principle. Wider application of this concept will transform transform business and personal life.

In the early days of the Internet, some communication experts remarked that there was theoretically only one network in the world, although some segments (subnets) hadn’t be connected into the whole yet. A similar thing can now be said about EPNs: there is theoretically only one EPN in the world, although some stove-pipes are not yet tied in – and some never will be. Nevertheless, the value of selectively broadening the scope of event processing is potentially enormous. Many years ago, Bob Metcalfe pointed out the geometrically increasing value of networks as they grow larger, a phenomenon called the “network effect.” The number of interconnections in a network increase in proportion to the square of the number of participants, and the value grows accordingly. A similar beneficial phenomenon occurs when different kinds of data from disparate sources are combined in an intelligent fashion using CEP (for real-time event data) and using other kinds of business analytics (for non-real-time data).

4.3 Holistic Event Processing

We believe that CEP will become a pervasive support technology in future business systems. But it will be largely invisible to the average user of those systems. These systems, essential to running every business, will be event driven systems employing all of the concepts of CEP. Just as the TCP-IP network protocols are critical to our Internet systems today, although few of us know anything about the workings of TCP-IP, so also CEP concepts such as event patterns, abstract events, event hierarchies, and event abstraction mappings, will be part of a technology arsenal employed in enterprise management systems. CEP will be used by technical support engineers and knowledgeable business specialists in customizing these systems to specific business issues. The average business user will see on-line graphical interfaces and other user-friendly tools that deliver mountains of event information in a human useable form. CEP is what makes it all possible, “under the hood”.

CEP will be an enabling technology in the development of very large scale information systems that form the foundation of our major application systems and business processes. Possible example systems that might come to pass are

• a global pandemic detection system,
• a world-wide unified air travel management system,
• a global food resources monitoring system,
• a system for monitoring the use of personal information,

and many other possibilities.

These kinds of future systems will consist of many collaborating smaller event processing systems. They will process not only large numbers of events, but large numbers of different types of events. Whereas a stock trading system processes a relatively small number of types of financial events (although it may process thousands of event per second), a pandemic watch system will process a huge number of types of events.

We call the kind of system that is composed of many smaller systems and processes very large numbers of different types of events a holistic event processing system. (i.e., the overall system is greater than the sum of its components)

A global unified air travel management system is one example that is already emerging by gradually piecing together existing air traffic control and airline management systems. It will take 20 or 30 years to fully develop.

Another example is a pandemic watch system for early detection of emerging infectious diseases. Small specialized systems for detecting SARS and other diseases have been operating for 20 years. The global pandemic system will evolve from smaller systems until it eventually process a huge number of types of events. This is likely to include, for example, mobile phone and text messages in multiple languages from the field in rural villages of SE Asia as well as from field agents of public health organizations, events input to social networks (e.g., rumor tracking), public health bulletins from local regional and national authorities, international news wires, pharmacy sales reports in all countries, as well as hospital and medical laboratory reports, airline passenger data and immigration data. The types of event input to this kind of system are many, and methods of dealing with inaccurate events are still to be researched.

Our point here is that holistic event processing systems are global in scale, and they are in our future. In fact we can see the beginnings of these systems now.

Holistic event processing systems will result by a process of evolutionary development, usually by haphazard extensions of existing systems and many trial and error experiments and failures. They will not be planned or designed or built as such. They will result from the political pressures for “right now” information and predictions from different departments within a company, or different communities within a nation. This is coupled with the need to meet economic constraints on the costs of developing new event processing systems. Instead of building new separate systems, old systems will be added to and extended beyond their original designs.

CEP will be an essential ingredient in the holistic event processing systems of the future.

5. Additional Reading

“Event Processing for Business,” David Luckham, 2012, John Wiley & Sons, ISBN: 978-0-470-53485-4. Available from many places including http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470534850.html

“Event Processing: Designing IT Systems for Agile Companies,” K. Mani Chandy, W. Roy Schulte, 2010, McGraw-Hill, ISBN: 978-0-07-163350-5. Available from many places including http://www.gartner.com/it/products/research/media_products/EP_book/index.jsp.