Data Doghouse - performance management, business intelligence, and data warehousing

People, Process and Politics - We All Hate Data Silos, So Why do They Happen?

Why are data silos so prevalent when we really ought to know better? We know that it’s risky to have inconsistent data. We strive for the “single version of the truth” along with our IT and high tech vendors.

So, with all this awareness, why does almost every company end up with data silos?

There are two causes that I’ve seen in my consulting work. First, companies typically organize BI and DW related projects on a tactical and project-by-project basis. This makes data silos very, very likely. Second, these tactical projects are designed without an overall information architecture blueprint. This absolutely, positively guarantees a silo. You “can’t get there from here” if you don’t know where you are trying to go.

There are several reasons why otherwise smart people create projects on a tactical basis without an overall architecture:

Funding - Projects are funded by a group representing an organization or business process. These groups are focused mainly on their own information needs and delivering business value in the context of those needs. Building an enterprise-wide solution is not in their minds nor in their charter.

Confusion - Industry jargon creates the impression that things are different, when they’re actually similar. There are many similarities between CRM (customer relationship management), SCM (supply chain management), budgeting and forecasting, performance management and balanced scorecards. Many people do not see that each has data, data integration and business intelligence that can be shared across projects.

Technology - People tend to treat different technologies as different projects and applications. ETL (extract, transform and load), SOA (service oriented architecture), EAI (enterprise application integration) and data virtualization are considered different and treated as separate entities rather than different methods to implement data integration.

Disconnects - Data warehousing and enterprise applications are treated as separate worlds. Even though the business person may need reporting from operational applications and the DW, different projects are created to implement reporting from each of these data sources. And often IT splits its ERP and DW people into separate groups, further perpetuating silos.

No Architecture  - Finally, without an information architectural blueprint how is anyone to know that these projects should be connected? If you split up building a house amongst various contractors and specialists such as electricians and plumbers without a blueprint and without these groups talking to each other it is unlikely the house gets built correctly. It’s the same with a DW or BI project. You need the blueprint (information architecture) - especially if the projects are set up tactically and on a project-by-project basis. This is called accidental architecture and it creates data silos.

Data silos result in poor quality data that is often inconsistent, wastes resources and time on overlapping and redundant projects, costs more money to build and maintain, and ultimately results in the business not getting information in a consistent, comprehensive and current manner. Data silos also encourage the creation of data shadow systems or spreadmarts, further exacerbating the problems.

How do we get out of this? Stay tuned for more posts in the People, Process and Politics series.

More than just “The Usual Suspects” - Doing Due Diligence for DW and BI Tools

The publishing of Gartner’s Magic Quadrants or Forrester’s Waves is eagerly awaited by vendors in many IT categories, particularly those who scope in the “top” quadrant or wave. I know when the latest data warehousing, data integration or business intelligence reviews are published because the companies who are category leaders start issuing press releases about their accomplishment. Congratulations, we are the best!

In our industry (DW and BI) the core group of products has remained the leader for many years. The companies have changed because of acquisitions, but the underlying products listed stay the same. The top two data integration or ETL (extract, transform and load) products and top three BI (business intelligence) toolsets are perennial winners. And because the market is mature, at least there is a core set of functionality (but there is much innovation and expansion possible), and the leaders pack has been expanding in recent years.

The annual results are getting boring with “The Usual Suspects” being the leaders every year. Sort of like an award show with the same singer or actor/actress winning every year, they may be the leaders in their field but…

This is not a “trash the reviewers” post. The work the industry research firms do is great and they provide an invaluable tool when you’re doing due diligence, especially if you are new to the market. However, these reports are often used as a shortcut to truly doing due diligence or as a hammer to reinforce a bias an IT group may have for a particular product. 

Misusing this research by no means invalidates it, nor suggests that the research methodology is inappropriate. Gartner even suggests how to use their research but, sadly, like warning labels on a pack of cigarettes, readers often ignore this advice.

In “How to Use a Magic Quadrant,” Gartner states:

“Your needs and circumstances should determine how you use the Magic Quadrant, not the other way around. To evaluate vendors in the Leaders quadrant only and ignore those in other quadrants is risky and thus discouraged. For example, a vendor in the Niche Players quadrant could offer functions that are ideally suited to your needs. Similarly, a leader may not offer functions that meet your requirements — for example, its offerings may cost more than competitors', or it may not support your region or industry. Use a Magic Quadrant to narrow your list of choices, but don't base your decision only on the model.”

I think the leaders in these categories are terrific. I have been involved in many successful implementations with these tools and worked with the vendors who sell these tools over the years. And the leaders’ products are not standing still, but continually being updated to incorporate expanded functionality and emerging technologies.

There are many companies that are only beginning their journey using DW, data integration and business intelligence tools. And, there are many more that are trying to shift from data shadow systems or spreadmarts to tools that will improve both business and IT productivity and enable the kind of reporting or analytics that their companies need. In this economy, budgets are tight and companies have to look at the TCO (total cost of ownership) and the amount of skills necessary to develop, deploy and support solutions. TCO is relative to your company size, but regardless IT and business need to look outside the usual suspects if for no other reason than to do the due diligence and determine what truly meets your needs.

A company may end up buying or expanding their use of leaders’ products, but at least they will have truly explored their options. There are very good products available from emerging firms; on-demand software or SaaS (software-as-a-service); open source software (OSS); and, smaller firms that have been around for years with many customers. There have been a quite a few mergers and acquisition (M&A) in our industry, but there are many firms that offer these products.

Podcast: BI tools vs. Microsoft Excel Spreadsheets

According to a recent survey, many users are still using Microsoft Excel spreadsheets to analyze and report on corporate data. These same users also spend much of their data-gathering time correcting out-of-date information in these Excel spreadsheets.

So what are the real benefits of using Excel spreadsheets to analyze and report data? What additional benefits do business intelligence (BI) tools offer?

In this podcast, Karen Guglielmo, special projects editor for SearchCIO.com, interviewed me about the differences between the two types of reporting tools. I also discuss how and if BI tools and Excel spreadsheets can be used together in the enterprise to gather and analyze data.

Listen to BI Tools vs. Microsoft Excel Spreadsheets

Finding Business Value in Data Warehousing/Business Intelligence During Tough Times

How are you spending your IT budget these days?

The stock market is down, housing prices are down, the cost of everything including food and oil is rising, there is a credit crisis, financial institutions are losing billions and going bankrupt, unemployment is rising and we are either in a recession or about to go into one (depending on who you listening to or reading.)

 

Businesses are responding by laying people off, canceling projects, cutting budgets and restraining expenses. Others may not be experiencing the downturn but are nevertheless being cautious on spending.

 

With this as a backdrop many people feel that they can’t undertake any new IT projects involving business intelligence, performance management and data warehousing. They think that you need significant investments in order to do these projects and to obtain business value. Also, software vendors, systems integrators and industry analysts seem to always associate these projects with significant investments. “You need to spend money to make money” is the adage.

 

Although you do have to make an investment to obtain business value from your BI/DW efforts, that investment may not require a huge capital outlay. In fact, you should leverage existing software and hardware infrastructure to avoid large expenses. You also probably don’t want to start a project that management will perceive as big and expensive during these economic times.

 

So where do you put your limited dollars where they will give you the biggest impact? One of the most fertile areas to invest in is renovating or replacing data shadow systems or spreadmarts. The business originally had valid reasons for creating these data shadow systems but, most likely, management is now irked by the time and expense that goes into their care and feeding. In addition, many business people feel that these systems are prone to error, and as they get bigger they get inflexible.

 

Too often IT’s reaction to the data shadow systems is a “big bang” project costing a lot of money. However, as I have discussed before, often you can revamp these systems with your staff’s “sweat equity,” business cooperation and maybe a little outside expertise to guide you on how to change your company’s approach to fixing this problem.

 

The good news about data shadow systems is that by definition they have business value. Otherwise business people would not have created them and maintained them. The bad news is that both business and IT people have to change their perceptions in order to revamp these systems and to get the business value out of them.

 

During tough economic times like these, it’s important to see data shadow systems for what they are and take advantage of any opportunity to make your IT budget dollars go further.

Dealing with Spreadmarts: Strategies for Migrating to a Managed BI Environment

I recently co-authored with Wayne Eckerson of TDWI the Best Practices Report and webinar: "Dealing with Spreadmarts: Strategies for Migrating to a Managed BI Environment."

Anyone who has read my articles and blog knows I often talk about data shadow systems. Read this report and listen to this on-demand webinar for the latest thinking and research on spreadmarts/data shadow systems.

About the Best Practices Report

In organizations all over the world, business people bypass their IT groups to get data from spreadmarts. Spreadmarts are data shadow systems in which individuals collect and massage data on an ongoing basis to support their information requirements or those of their immediate workgroup. These shadow systems, which are usually built on spreadsheets, exist outside of approved, IT-managed corporate data repositories, such as data warehouses, data marts, or ERP systems, and contain data and logic that often conflict with corporate data. Once created, these systems spread throughout an organization like pernicious vines, strangling any chance for information consistency and reliability. You’ll find them in all industries, supporting all business functions. According to TDWI Research, more than 90% of all organizations have spreadmarts.

The technical remedy for spreadmarts is to manage and store data and logic centrally in a uniform, consistent fashion and then let individuals access this data using their tools of choice. At TDWI, we call this a managed business intelligence environment. The goal is to transform spreadmarts into managed spreadsheets. This lets IT do what it does best—collect, integrate, and validate data and rules—and lets business analysts do what they do best—analyze data, identify trends, create plans, and recommend decisions.
Download the Report

About the Webinar

One of the most insidious problems in data warehousing is the proliferation of spreadmarts—spreadsheets and Access databases that masquerade as data marts. Created independently, these data shadow systems deliver a fragmented view of the enterprise and an inconsistent view of the business that cripples the business. This Webinar will show how spreadmarts are a symptom of a larger problem and then outline strategies for migrating organizations to a managed BI environment that delivers a single version of truth.
Watch this TDWI Webinar

(Click for more business intelligence webinars)
 

TDWI Spreadmart Research Report

I'm working with Wayne Eckerson (Director, TDWI Research,  The Data Warehousing Institute) on their new Best Practices Research report, "Strategies for Managing Spreadmarts and Integrating with Microsoft Office," which will be published in January 2008.

If you take the survey, you can be among the first to receive the report.

One of the biggest pain points BI professionals face is dealing with "spreadmarts" - renegade spreadsheets and desktop databases that suck up expensive analyst time and undermine data consistency and compliance efforts. The question is, how do you enable users to work in Microsoft Office applications without undermining a single version of truth? Our upcoming report will answer this question and provide several approaches for corralling renegade data shadow systems and explain technical advances for integrating with Microsoft Office applications.

To write this report, we need your help. We'd like to understand the scope and costs of spreadmarts and effective remedies. Please let us know how you are approaching spreadmarts by taking our 15-minute survey at the following Web address:

http://www.tdwi.org/info.aspx?id=42175

For completing the survey, you will be the first to receive the final report when it is published in January 2008. If you encounter difficulties, contact Wayne at   weckerson@tdwi.org.

Ouch! Get Ready — Spreadsheets Are Here To Stay For Business Intelligence

I’m not the only one who thinks that Microsoft Excel spreadsheets are here to stay. (See Microsoft Excel: The King of BI.)

In "Ouch! Get Ready — Spreadsheets Are Here To Stay For Business Intelligence", Forrester analyst Boris Evelson has similar thoughts:

"Spreadsheets — the most widely used business intelligence (BI) tool — are a permanent fixture in enterprises because no other analytical application outperforms them in flexibility, ease of use, and ubiquity. Spreadsheets' role in BI is no longer limited to simple import/export mechanisms; they now play an integral role in all layers of the BI stack. Yet the lack of controls and security and integrity issues create tremendous challenges. To minimize risks while gaining the inherent BI value of spreadsheets, information and knowledge management professionals must discriminate between the different ways spreadsheets are used. Then, they must help users apply advanced spreadsheet tools and techniques to their daily jobs, while also implementing a tightly controlled (or closely monitored) environment for critical production processes that rely on spreadsheet data. In turn, vendors should take advantage of this market opportunity by introducing tools that will bridge the gap between spreadsheet management and spreadsheet usage in the BI process."

What I really find refreshing about this analysis is that is does not offer a silver bullet solution that your often find in vendor-sponsored white papers. The report states that "Although spreadsheets introduce numerous management issues and risks into the business, both IT and the business must accept that eliminating spreadsheets is not a realistic — or even desirable — option."

All the solution alternatives include people, process and products. Data may not be able to be controlled given the high tech capabilities that are at people's fingertips BUT it can be managed. If data is to be treated as the corporate asset it is, people have to assume their responsibilities which are supported by corporate processes and enabled by their product infrastructure.

Maybe you feel spreadsheets are from hell or maybe you think they are a godsend (a Mars vs. Venus or IT vs. business perspective) but it does not matter what you think...spreadsheets are here to stay (at least for awhile.)

The complete Forrester report is available here. (free, but registration required)

Oracle User Group Survey: Achieving End-to-End Business Intelligence

It was interesting to read a recent article from DMReview.com “OAUG Survey Reveals Three Out of Four Enterprises Using Oracle Applications Still Use Spreadsheets for Business Intelligence Reporting”. The findings summarized were from the survey "Achieving End-to-End Business Intelligence", conducted by the Oracle Applications Users Group (OAUG) in order to gauge the depth of BI and analytics deployments within Oracle enterprises.

3 out of 4 enterprises rely on (my emphasis) on spreadsheets for reporting and business intelligence! A corollary finding was that only one of ten employees in a corporation had access to BI capabilities (outside of Microsoft Excel, of course.)

Does this surprise anyone? If it does then you probably have not talked to many business groups in corporations today. And I do not mean just the SMB (small to medium size business) market but also the Fortune 100. And if you are surprised you probably have not read about data shadow systems.

 The survey found that 4 out of 10 respondents reported that their departments generate more than 100 reports a month taking a great deal of time to create, maintain and, of course, modify to meet changing business requirements. More than 60% said it took a day or more to generate a single report for their business users.

The survey also found that people were worried about the data quality in these reports.

Since the survey was of companies using Oracle applications, some people (maybe Oracle’s competitors?) may say that this is just a condition specific in Oracle shops. This condition is NOT unique to Oracle applications. It is pervasive throughout companies big and small, whether they have Oracle, SAP or many other business applications. Microsoft Excel is the pervasive BI tool at this point.

The biggest concern any enterprise should have is the condition of the data that is being used to make decision rather than the BI tool used. It is great to have a terrific and sexy BI tool that can slice and dice data any way a business user can image examining the business, BUT if the data is just then “garbage in, garbage out”.

And once the data is in order, preparing the data for information understandability and consumption is the next task. Just purchasing hundreds (or thousands) of licenses for a “true” business intelligence tool for your enterprise does not guarantee these products will be used or that you are providing business value. If that was true, then 3 out of 4 companies would not be relying on Microsoft Excel for reporting and business intelligence.

The Data Shadow System Conundrum

As Martha Stewart would say, getting information into the hands of business users is a "good thing." Every enterprise needs to provide accurate, timely information for business groups to fuel the operational decisions they make each day.

So does it matter how they get this information? Business users may wonder what difference it makes as long as they get the information they need every day to keep their business selling products and services, working with suppliers and partners, and competing in the marketplace.

Well, it does matter how business users get information. It matters because, quite often, they get it from data shadow systems, which are groups of spreadsheets and local, customized databases - often Microsoft Access and statistical databases - created by business groups to gather data for their users. While these systems provide exactly the information that business users are asking for, they are rarely part of an enterprise's official data warehouse corporate performance management strategy. Outside the purview of the IT group, they often spawn data silos with the usual problems of inconsistency and quality.

Read more in my DM Review article: The Data Shadow System Conundrum