Avoiding Disasters Waiting to Happen

Managing the real time enterprise, grand visions for the future of eCommerce, and RFID, what have they all got in common?

Describes the need for a new technology to manage real time enterprises. Introduces the concept of IT blindness. Gives examples of IT blindness in electric power transportation, on-line banking, industrial automation and retailing. Describes new eCommerce visions (instant insight, web services, anticipatory supply chains) that depend upon solving IT blindness. Describes how new technologies often need other new technologies in order to be useful with RFID as the example. All of these problems, managing in real time, grand visions for eCommerce, and new technologies like RFID depend upon solving IT blindness…

Corporations are becoming increasingly real time by utilizing both private networks and the Internet. What we are witnessing is the emergence of the “event-driven real time enterprise” as Gartner calls it, in which the whole business structure, processes and applications are event-driven. While Internet-based automation helps streamline businesses, cut costs and make them more profitable, it introduces a new problem – managing in real time.

Managing the real time enterprise is becoming an increasing challenge. This is amply illustrated by the North Eastern power blackout in August 2003. As Congressional hearings uncovered, no manager had a global view of that event-driven situation. Here was an example of a collaboration between a dozen electric power companies across several states and Canada, aided by federally mandated independent service operators (ISOs). The business processes of these companies, in which humans are very much part of the process, collaborate over the power transportation grid, SCADA control systems, and telephones. In a period of four hours a sequence of grid events such as power lines failing and generation units disconnecting, built up to a cascading blackout across the Midwest, the Northeast and parts of Canada. The regional ISOs took no active steps to stop the progression, largely because they had no global overview of the events that were happening. And no understanding of the significance of the events their monitors did show them. The New York Times commented: “the ISOs were like air traffic controllers trying to keep order in the sky without knowing where all the planes were.” Meanwhile power company business managers discussed high level sales strategies by phone which they did not know would create grid events that only worsened an already critical situation. This was IT blindness, big time.

IT Blindness.
The problem with managing real time operations is that enterprises don’t know from a business perspective what is going on in their IT infrastructures, minute by minute or day by day. Technology for monitoring IT traffic is stuck at the network level which is useless as business intelligence. Managers can’t figure out in real time when events on their IT layer – and it’s not just single events, but more generally patterns of many events – are significant from a business perspective. This is IT blindness. It’s a real time problem. It can’t be solved by storing events in databases and searching them later – that just isn’t fast enough.

Almost every day a quick search of the internet, or the news media, will reveal a new example of IT blindness, an enterprise that didn’t recognize the importance of the events on its IT layer until a catastrophe happened. Example. A banking institution didn’t know that its on-line customers were behaving oddly until it was informed – by email from an astute customer – that its on-line banking website was being “phished”. Customers’ identities had been stolen. By then the thieves, using customers’ identities, had installed new automatic payments on their accounts, something the real customers never did before, and transferred money to mole accounts. The publicity makes headlines! And the bank makes good its customers’ losses.

A famous example of IT blindness is the Australian sewage disaster. This happened a few years ago, but is indicative of what can happen today, any day, to our nuclear power stations, dams, and power grids – as Rickard Clarke would tell us! A disgruntled ex-employee of a software company gained access electronically to the control computers of a state-of-the-science sewage treatment plant on Australia’s Sunshine Coast. He released millions of liters of raw sewage into a protected sea inlet, corrupting the local environment for many years to come. Yes, he broke in 46 times before he was caught – and that was purely by luck. The plant’s IT management certainly knew they had a problem. But they never detected events from a spurious controller communicating with their system by wireless. They simply didn’t know how the spills were occurring. The FBI’s National Infrastructure Protection Center (NIPC) issued an advisory in the light of this disaster, “control and telemetry systems should be monitored for possible trends that may evolve into malicious activity.” The question is how — NIPC didn’t tell us that.

You might well ask, “what have such catastrophes to do with normal day-to-day business”? Of course, what hits the press these days are only the sensational and egregious examples of IT blindness. Most examples are not catchy enough to make headlines. But they happen every day. A global retailer’s pricing process sends out incorrect pricing data to 700 stores, and for a few days millions of dollars are lost until the error is caught. That’s not headlines, but it is IT blindness, and it is bad for the profit margin! The business processes of any enterprise that operates competitively these days are at similar risk of strange and costly behavior. And, if you don’t move towards eBusiness, you’ll be out of business.

Obviously, we have to solve IT blindness to manage our businesses better than we do!
A new technology is needed that allows business process managers to understand the event activity in their information systems. They cannot decipher reams of network logs, nor do they have the time to do so. They need high level views of event activity as it happens, views that immediately indicate how critical business functions are being impacted.

My next article describes how grand visions for the future of eCommerce, and some of our newest technologies such as RFID, also need a solution to IT blindness.

© David Luckham 2004