Re-Energizing BI with BPM

by kirangarimella on April 1, 2007

Years of experience with data and information have taught our industry the value of business intelligence (BI). Most executives appreciate the need for BI and are not averse to funding BI initiatives.

However, they continue to be frustrated with the lack of support for decision-making. They express little confidence in the reports that the BI solutions provide. Arguments ensue about not just the semantics of the data, but about the quality of the data itself.

A saying is attributed to the former U.S. Senator, Daniel Patrick Moynihan, “You are entitled to your opinions, but you are not entitled to your own facts.” Unfortunately, BI does not prevent its users from creating different versions of the truth.

Additionally, BI solutions are designed from a functional perspective; at best, they have a tenuous link with the underlying business processes that flow across functional boundaries. This hinders users from correlating BI with end-to-end business processes.

Analyzing the Problem

Two causes contribute to this sad state of affairs: BI creates an artificial barrier between transactional (OLTP) data and historical (OLAP) data, and it does nothing to prevent a tight coupling between applications and data.

People analyze events and make decisions based on what happened in the past, what is happening right now, and what is likely to happen in the future. In the normal decision-making process, people do not put up artificial barriers between these time domains. Consumers of BI need to traverse between data and information in the past, the present and the future.

An architecture that is hampered by poor integration infrastructure cannot facilitate a tight coupling between the OLAP and OLTP domains. Traditional BI downplays the need for real-time intelligence because it is considered statistically insignificant for analytical purposes.

This combination of technical constraints, piecemeal solutions, and lack of attention to business processes divides the natural analytic process into three distinct subprocesses that do not flow into or connect with each other.

Instead of analyzing the past, the present, and the future in a seamless way, users are forced to move between tools, processes, and methodologies to deal with temporally segregated data. BI solution providers further limit themselves by asking their users what they want, rather than asking them what they want it for.

By dealing predominantly with the past, BI is relegated to the role of a fancy aggregator of historical data rather than a provider of true, comprehensive intelligence.

With the increasing focus on execution, rapid change, and forward thinking, the past is becoming irrelevant.

Operational reporting, which focuses on the immediate past (defined as real time), is outside the domain of traditional BI. Forecasting and predictive analytics, the domain of resource planning systems (such as manufacturing resource planning) or other specialized tools, are not as well known or adopted as historical BI.

Therefore, we have a paradoxical situation where traditional BI (i.e., polishing the rear-view mirror) is an excellent but irrelevant solution, while there is not yet an alternative, compensating focus on the more important operational BI and predictive analytics (i.e., cleaning the windshield).

Common Application Architectures

The other contributing cause to dysfunctional decision-making is the common enterprise architecture model that tightly couples applications and data. The characteristics of this type of application architecture are:

  1. Applications own the data. Data that is captured within the application ends up being stored inside that application. As a result, data islands are created. There is no relational linkage between these application-specific data repositories.
  2. Every application creates its own reporting functionality. The reason is that applications are designed as if they are shrink-wrapped systems that should be self-contained. This may be fine for software that you buy in a computer store, but there is no need to design enterprise applications as if they will be sold on a shelf.
  3. There are no common semantics. The same fields or similar fields may have either different data types, different sizes or a different set of legal values. As a result, BI users interpret data according to their own semantic understanding.
  4. Data is semantically corrupt. Data fields take on values that corrupt their original purpose. Some of this is caused by the inflexibility of the original application. At other times, it is caused by a lack of intellectual rigor.

So, BI erects a barrier where none should be (i.e., between historical and actual, between viewing and doing) and is powerless to enforce a barrier where there should be one (i.e., between applications and data, which should be very loosely coupled).

Usual Solutions are Inadequate

The traditional approach to BI is characterized by its overwhelming focus on data.

It reminds me of the story of the rebellious teen who was frustrated with his rich father’s preoccupation with money and his apparent lack of social conscience.

“Isn’t there something more important than making money?” he asked.

To which his father replied, “Yes, certainly. Counting it, saving it, investing it, and spending it.”

Similarly, BI users may well ask their IT colleagues in exasperation, “Isn’t there something more important than just data?”

To which the normal BI professional replies, “Yes, certainly. Collecting it, cleaning it, storing it, transforming it, aggregating it, and presenting it.”

All of these data-focused activities are important and essential.

However, a compilation and summarization of data will not automatically yield intelligence any more than a heap of auto parts can carry you from point A to point B. When users download BI reports onto their desktops and use Excel to perform pivot table analysis on them (even though their BI platforms may provide some drill-down and pivoting capabilities), it shows that BI solutions are unusable without relying on some external crutches.

The focus on data in and of itself would be fine if BI were instead called data aggregation and presentation – but even then it may be an expensive solution.

Enterprise data modeling and management (EDM) is the usual response to the problem of data islands. The central idea of EDM is correct. However, it is an infeasible approach because it addresses the symptoms and not the root cause of the problem – poor application architecture, which is itself caused by a lack of process thinking. Some of the specific challenges for EDM are:

  • It presupposes that all constituents agree to one common definition of data. This is unreal not because people willfully disagree, but because they are accountable for holding different perspectives.
  • It falsely assumes that employees have the time to lay aside daily operations and work on creating a shared understanding.
  • The solution takes too long to implement; momentum is usually lost or the business itself changes while the solution is being designed.
  • If (and when) the EDM solution is in place, it needs to be constantly monitored and maintained to ensure that it does not get out of sync with business reality.
  • EDM’s artifacts (such as data dictionaries) are usually directed toward a technical audience. These components are not particularly friendly toward the business user and, as a result, fail to become part of their toolset.

Exorcising BI

BI is, by definition, the intelligence about the business. Business processes are the fundamental operational artifacts of business. A proper approach to BI, therefore, begins with the realization that BI is gathered and used in the context of business processes. The consumptive and analytic units of BI include data, rules, documents, organizational structures, roles, key performance indicators, metrics and process models. Some of these BI units originate in business processes and are captured by the associated applications, others are linked to business processes or they are defined and stored in them. A business process management (BPM) platform enhances BI by supplementing it with new types of analytics and superior governance capabilities. By using a synergistic integration of BPM and BI:

  • Users will be able to relate BI to business processes.
  • IT architects will be able to relationally link data islands together and offer seamless, end-to-end process visibility to business users.
  • Business analysts will be able to investigate and document reporting needs as part of the project’s requirements-gathering phase. They will be empowered to ask their business customers what they want the reports for and to design the presentation of information in a way that obviates the use of additional, standalone analytical tools.
  • Business users will be able to convey their decision-making and analytic needs to IT rather than just requesting a compilation of data.
  • Project managers will be able to institute checkpoints throughout the project lifecycle to ensure that data gets the respect it deserves.
  • Data stewards will be able to monitor and resolve semantic corruptions and exceptions in business data, thus ensuring very high quality of data.
  • The systemic collaboration between BI and BPM provides unparalleled support for decision-making, which is the raison d’être for BI. Specifically, it ensures that users have trust in the quality of data, the reported metrics are aligned to business priorities, the conclusions derived from the data are logically sound and provable, the information exposes clear courses of action, users have visibility into the business processes and users can make timely decisions.

BI does not offer true analytic and decision-making capabilities to the extent that it does not provide time-transparent analysis, linkage to business processes and data governance. Two architectural flaws that create and exacerbate the problem are the artificial barrier between historical and operational data and the tight coupling between data and applications. A data-focused approach to BI may at best yield good data aggregation, but does not provide true BI. EDM, a common response to data governance, is a heavy-handed solution that may not be sustainable for most companies.

A better approach is to integrate BI and BPM. By providing various supplementary capabilities (such as process modeling, process analysis, metadata management and governance, operational metrics, process management, business activity monitoring and service-oriented enterprise architecture), BPM breathes new life into BI.

[First appeared in DM Review (now, Information Management).]

Leave a Comment

Previous post:

Next post: