Why Organizations Use Event Stream Processing With AI

W. Roy Schulte

21 February 2025

An increasing number of real-time business applications use AI in combination with event-stream processing (ESP) software (sometimes called complex-event processing (CEP) software). This is natural because AI is effective only when it has good, relevant input data.  Real-time AI requires real-time input – data about what is happening now. Not all real-time data is streaming data, but a lot of it is (and, sure, some historical information is usually relevant too).

Event streams support situation awareness, which means knowing what is going on in the moment so you can make timely decisions about what to do. Situation awareness matters regardless of whether decisions are made by people or by machines. An event stream is an unbounded, continuous sequence of data records (event objects) that report things that happen (event happenings). Common examples include clickstreams, sensor data flows, data broker feeds (e.g., containing news, traffic, weather, or market data), machine logs, copies of business transactions, social computing activity streams, and similar data flows. Every medium size or large enterprise has many thousands of different event streams continuously flowing over its networks.

ESP is often implemented without using AI tools, and AI and other analytics are often implemented without ESP tools. However, using them together provides the best available solution for an increasing number of business problems because of the high information value and ubiquity of streaming data. ESP and AI software tools are complementary in three ways:

• AI tools can make operational ESP business applications smarter at run time by applying a wide variety of advanced mathematical techniques, sometimes now even including Generative (Gen) AI transformer models.

• AI-based copilots can make it faster and easier for software engineers to develop and test ESP applications, regardless of whether the applications use AI or other analytical techniques at run time.

• Conversely, ESP can be used to implement streaming data engineering pipelines that prepare event streams for use by engineers as they design, build, and train AI and other analytical solutions.

Vendors offer a wide variety of software that supports ESP/CEP. The most obvious are ESP frameworks like Flink and Spark Structured Streaming. But ESP is also performed in other kinds of products including Data Streaming Platforms (DSPs), Unified Real-time Platforms (URPs), Stream Data Integration (SDI) tools, streaming DBMSs, streaming-enabled data platforms (lakehouse platforms), and some SaaS and packaged application solutions.

For a deeper analysis of products and the relationship of AI and ESP, see our two-part series, “Event Stream Processing Helps AI and Vice Versa”. Part 1 explains the first two of those bullet points, describing now AI helps ESP systems. Part 2 explains how ESP facilitates the development of AI models by enabling streaming data engineering pipelines..

Streaming data engineers need to understand AI, and AI engineers need to understand event streams and the technologies for ESP, particularly when addressing real-time operational business problems. AI and ESP are mostly orthogonal. They use quite different mathematical techniques but their history overlaps in ways you may not realize. Dr. David Luckham is a key figure and pioneer in the field of ESP/CEP, particularly for his work on the Rapide project at Stanford University in the 1990s. His 2002 book, “The Power of Events,” introduced the term “CEP.” It was the first explanation of modern CEP concepts, and is still widely referenced. But years earlier Dr. Luckham was a member of John McCarthy’s AI team, first at MIT and later at Stanford. McCarthy is, of course, often called the father of AI because he coined the term “artificial intelligence” and developed the LISP programming language. At MIT, Dr. Luckham helped develop an early LISP compiler. His later AI work at Stanford included reasoning systems for proving mathematical theorems before moving on to his CEP innovations.