Data Doghouse - performance management, business intelligence, and data warehousing

The Scourge of Data Silos

"Those who cannot remember the past are condemned to repeat it." [1]

Over the years there have been many technology waves related to the design, development and deployment of Business Intelligence (BI).  As BI technologies evolved, they have been able to significantly expand their functionality by leveraging the incredible capacity growth of CPUs, storage, disk I/O, memory and network bandwidth. New technologies have emerged as enterprises’ data needs keep expanding in variety, volume and velocity.

Technology waves are occurring more frequently than ever. Current technology waves include Big Data, data virtualization, columnar databases, BI appliances, in-memory analytics, predictive analytics, and self-service BI.

Common Promises

Each wave brings with it the promise of faster, easier to use and cheaper BI solutions. Each wave promises to be the breakthrough that makes the “old ways” archaic, and introduces a new dawn of pervasive BI responsive to business needs. No more spreadsheets or reports needed!

IT and product vendors are ever hopeful that the latest technology wave will be the magic elixir for BI, however, people seem to miss that it is not technology that is the gating factor to pervasive BI. What has held back BI has been the reluctance to address the core issues of establishing enterprise data management, information architecture and data governance. Those core issues are hard and the perpetual hope is that one of these technology waves will be the Holy Grail of BI and allow enterprises to skip the hard work of transforming and managing information. We have discussed these issues many times (and will again), but what I want to discuss is the inevitable result in the blind faith in the latest technology wave.

Common Results

Each wave typically advances BI in some way, but does not really create the breakaway promised; truly pervasive BI remains the Holy Grail. What is almost guaranteed with each new wave, however, is yet another data silo added to the many that already typically litter an enterprise landscape. As a result, many enterprises have one or more of the following: multiple data warehouses, operational data stores (ODS), independent (silo’d) data marts, spreadmarts, data shadow systems and scattered databases used for reporting. Further evidence of the tech wave spawning silos is that enterprises typically have multiple BI product implementations with six or more different BI products.

The people involved in each new wave typically assume that their predecessors (who implemented a previous tech wave) did not know what they were doing and/or just did not use the right technology. They think they know best, and the previous generation (maybe just a few years from being the latest wave) has nothing of value to teach them. Hence, the newest generation does it all from scratch, encountering the same pitfalls – pitfalls that often have nothing to do with technology and everything to do with data management, information architecture and data governance.

Sometimes, they create a new solution separate from the existing BI environment. After all, why get bogged done in the old wave, now a legacy application, when you can start from scratch. This means the next tech wave builds a new tech silo. It may produce business value in the short run, but in the long run the business group and IT have another silo to reconcile.

Enterprises need to look at their Big Data, data virtualization, columnar databases, BI appliances, in-memory analytics, predictive analytics, and self-service BI efforts and determine if they are going to be the next data silos. Unfortunately, the answer is likely to be yes.

The bottom-line is that business people need data and analytics from across the enterprise and not just from a new shiny BI silo. But for this to happen, the BI team has to stop the madness by not creating yet another silo and legacy application. Can you say “no” to the data silo?

 

---

[1] George Santayana (1905) Reason in Common Sense, volume 1 of The Life of Reason

Report Governance is Key to Consistent Business Information

Much has been written about managing an enterprise BI Program and implementing data governance. Both of these are key ingredients to business value, but even with well-run BI and data governance programs, business people can get different results when they access different BI deliverables.  When this happens, they have to either reconcile or debate the differences. Either way, they’re wasting their time massaging data rather than using it to make business decisions.

BI deliverables need solid report governance in order to provide consistent information with which the business can make decisions. Report governance includes not only reports but also dashboards, scorecards, self-service BI, ad-hoc query, OLAP (on-line analytical processing) analysis, predictive analytics, data visualization, data mining and spreadsheets along with the data used.  It is more accurate to refer to this as “analytical governance” rather than just report governance.

Debating the numbers

Why do enterprises keep encountering these problems even when the data is fairly well managed from source systems to their enterprise data warehouse (EDW)? The reason is a lack of consistency: after data leaves the EDW it gets filtered and transformed, and then and different business rules are applied in the various BI objects. As tempting as it may be to blame this on business people who manipulate data in their own spreadsheets, the reality is that IT is just as culpable. This is because different IT people develop reports for different business groups independently.

What starts out as consistent data goes by the wayside because that data is manipulated in countless ways before the business people see it in various reports, dashboards or spreadsheets.  And if the enterprise does not have data governance discipline the problems are even greater.

Further compounding the problem is that a large enterprise typically has from six to nine BI products  -- each one doing things differently.

What to do and what traps to avoid

Typically IT reacts to the above conditions by:

• Establishing a very arduous BI design, development and deployment process.
• Trying to lock down reporting by making it difficult for business people to create spreadsheet reports that pull data from various data sources
• Consolidating the enterprise to a single BI tool

All of these are bad ideas.

Building up an arduous BI process or trying to limit the use of spreadsheets goes completely against the primary objective of BI, i.e. to enable the business to access data, analyze it and act upon the analysis. After all, what good is data if it is not used for a business purpose?  And in the absence of comprehensive and consistent BI, business people will fill-in the vacuum with spreadsheets and spreadmarts or data shadow systems. IT should have no illusions that they can (nor should) prevent the business from using spreadsheets.  Business people have to make decisions whether they have the data or not, so it is better that they have it.

Although consolidation onto a single BI tool seems appealing, the reality is that is becomes a very long-drawn out political battle to select one tool and then a long-drawn out migration that initially only delivers the same report only in different tool. The ROI for this initiative is weak at best and too often fails to get to one tool since typically the other BI reports are left as legacy (and development continues in them regardless of what people claim.)

Next…. What to Do and The Business Value

BI M&A Continues with a Twist: QlikTech Acquires Expressor

 QlikTech (QLIK), a leading independent business intelligence (BI) vendor, announced today it has acquired expressor software.

QlikTech, with over $300 million in sales, was recently listed by Forbes as number 3 in their list of the fast growing tech firms “The Forbes Fast Tech 25: Our Annual List Of Growth Kings”. QlikTech’s QlikView is a BI product using in-memory data store to enable a business (or data) discovery platform for its customers.

expressor has developed an innovative data integration platform with a semantic ETL layer offering data and metadata management. The two companies formed a strategic partnership during the last year with the QlikView Expressor Server.

expressor’s data integration and management software will fill out QlikView’s capabilities and support enterprise data governance. This expanded functionality should enable QlikView to expand into more enterprise BI environments.

Mergers and acquisitions (M&A) have been a constant part of the BI landscape for more than the last decade. The high tech titans – IBM (IBM), Oracle (ORCL), Microsoft (MSFT), SAP (SAP) – have collectively acquired dozens of software firms in the DW/BI industry over that time frame to build and expand their product offerings. They continue to acquire firms to either fill in some gaps in their offerings (tuck-ins) or to incorporate some of the innovative ideas developed by start-ups or smaller independent firms.

QlikTech’s partnership and now acquisition of expressor points to a new wave of interaction between smaller independent BI software firms. Previously these firms would try to partner with the high tech titans to “ride their coattails” and sometimes hope they would be acquired by them. Now it seems that the smaller firms are aware they can partner or acquire peers that have complimentary solutions.

Although BI is often seen as a mature market there are exciting innovations in data visualization, in-memory analytics, Big Data, mobile BI, BI appliances, cloud BI, predictive analytics and BI embedded in operational workflow.  The business value of BI is significantly expands as the data that enterprises can use to make more informed decisions explodes.