Why Organizations Undertake Event Processing

Roy Schulte, David Luckham

16 November 2023

Event processing can be complicated and expensive, at least when you are new to it. Nevertheless, most organizations are expanding their use of event processing by means of three different mechanisms: basic event broker messaging (e.g., using Kafka or its counterparts); stateful event stream processing (ESP, e.g., using Flink or its counterparts); and various other software tools and applications that process event streams. To explain the trend, here are four factors that impede the adoption of event processing and four factors that motivate more event processing.

Impediments to event processing

Event processing requires expertise in at least two, and sometimes three subjects:

1. Event-driven architecture (EDA)

Application developers universally understand synchronous, request/reply application architectures, e.g. using subroutines and APIs, but many lack experience with EDA. EDA requires asynchronous design patterns, often involving publish-and-subscribe messaging communication. Developers also may need to implement message transformations or canonical message schemas to make the messages relevant to multiple consumer (recipient) applications.

2. Messaging middleware

The software infrastructure for conveying messages between producers (senders) and consumers is new to many development teams. Kafka is complicated because it uses multiple clusters and requires careful design to achieve qualities of service such as reliability and recovery from failures; exactly-once message delivery; security; load balancing; high throughput; and low latency. The same challenges also arise with alternative event broker messaging products, such as Amazon Kinesis Data Streams, Microsoft Azure Event Hubs, Pulsar, RabbitMQ, Solace PubSub+, Synadia Communications NATS, TIBCO Messaging and others, although each of these takes a different approach to the issues. With any messaging product, it takes time and expertise to configure, run, and maintain these subsystems, especially for on-premises or private cloud configurations. For the many companies that already have an event broker in production, additional challenges related to scaling up to handle many topics, clusters and applications emerge.

3. Event stream processing (ESP)

A growing number of streaming applications need to perform stateful ESP (also called complex-event processing, CEP) on sets of multiple events from one or more event streams. ESP may be used to support operational business applications, analytics, or stream data integration (i.e., landing event streams into databases, which involves varying levels of transformation). These tasks require ESP skills in addition to EDA and messaging middleware skills. ESP can be custom built into an application, or it can be implemented on top of an ESP platform such as Flink, Kafka Streams, ksqlDB, Microsoft Azure Stream Analytics, SAS ESP, Spark Streaming, Software AG Apama, TIBCO Streaming, or other products. ESP developers encounter issues that don’t appear in conventional synchronous applications or simpler EDA messaging systems. For example, they may have to deal with events that arrive out of order, missing data, misformed events, and clock disparities in distributed topologies.

The fourth impediment is a business issue that goes beyond the expertise issue:

4. Event broker messaging and ESP products are incremental expenses.

Event brokers and ESP platforms are incremental expenses on top of the DBMSs, API managers and other infrastructure software products that organizations already have. There are budgets to justify and selection and procurement headaches. It is often still difficult and expensive to find architects and software engineers with experience in event brokers or ESP platforms. There is a perception that working with data in real-time costs a lot more than performing the same business function with nightly or weekly batch runs. Managers sometimes ask “Why do I need event streaming now when I didn’t need it in the past?”

Factors that drive adoption of event processing

Organizations are expanding their use of event processing because none of those four impediments are actually compelling.

Consider first the expertise issues:

1. EDA is not rocket science.

The challenges of EDA are mostly confined to an initial learning period – software developers generally become proficient fairly quickly. Moreover, developers have been dealing with some event-driven, asynchronous processing concepts in graphical user interfaces and other realms for decades.

2. New software tools and cloud-based messaging Platform as a Service (PaaS) offerings greatly reduce the burden of supporting messaging middleware.

Several dozen vendors offer development, monitoring and management tools for Kafka and other event brokers, simplifying the tasks of implementing and managing event brokers that run on-premises or in private clouds. Furthermore, more than 20 vendors offer cloud-based managed services (PaaS) for event brokers and ESP platforms. PaaS offloads most of the work for running the infrastructure from user organizations’ platform teams. Configuring, deploying, monitoring, scaling up, performance tuning and trouble-shooting are significantly easier than they were only two or three years ago.

3. ESP development facilities are maturing.

ESP platform vendors have introduced high level development facilities so the developers can use SQL, Python, other high level event processing languages, or stream design tools instead of lower-level programming languages (such as Java or Scala). Developer still must understand the principles of stream processing, but design and coding are simpler and faster than in the past.

However, the real reason that organizations are doing more event processing is a business issue: If your competition doesn’t have it, you have a competitive advantage. If your competition has it, you’d better have it too!

4. Event processing provides business value by enabling better business decisions.

Event streams carry context information about events and the state of the business. Every company already has vast amounts of streaming data – clickstreams from web sites; call logs and emails from customer contact centers; other interaction data; copies of business transactions; location data from mobile devices and vehicles; sensor data from machines and buildings; and much more.

Life and business are intrinsically event driven. Situations that require near-real-time decisions occur in a multitude of business processes:

• Organizations have to make real-time next-best offer decisions whenever a potential customer clicks through their e-commerce web site.
• Banks have to make real-time fraud risk decisions whenever a customer requests an ATM transaction or presents a credit card to buy something.
• Fleet managers have to make immediate decisions to re-route trucks when traffic conditions change or equipment breaks down.
• Car ride services have to match passenger requests with driver and car availability.
• Telecommunication companies have to make real-time decisions to support vast networks with millions of mobile subscribers and tens of thousands of cell towers, switching stations and other devices.
• Airlines need to reschedule flights, crews, airport gates, maintenance and other activities as soon as weather, maintenance or traffic disruptions occur (See recent Southwest’s meltdown).

All of these applications rely on streaming events to make good decisions – APIs, file transfers and shared databases can’t provide the required fast response times and scalability.

The business case for event processing needs to consider more than just the cost of the labor, software and hardware. It needs to weigh the hard financial benefits of better real-time decisions in those, and many other, business situations:

• incremental sales and profit when customers accept more offers
• reduction in the amount of fraudulent financial transactions
• cost savings in truck driver labor and vehicle expenses
• increased revenue due to improved car ride passenger experiences
• decrease in customer churn based on increased telco network reliability and performance
• more efficient scheduling of resources and better airline passenger service

When all costs and benefits are weighed, event processing is a net plus, not an incremental expense in a growing number of business situations.