Data Doghouse - performance management, business intelligence, and data warehousing

Data Integration: Beyond ETL and Data Warehousing

Data integration suffers from an image problem. It has become synonymous with extract, transform and load (ETL). Likewise, ETL has been regarded as a data warehousing (DW) technology.  Both of these viewpoints do not reflect current capabilities and greatly inhibit enterprises in their attempt to integrate data to provide the information their business needs.

Because of this short-sightedness companies have lost opportunities to harness information as a corporate asset. It increases the cost of integrating data, encourages the creation of data silos and forces business people to spend an inordinate amount of time filling the information gaps themselves through data shadow systems or reconciling data.

>>> Continue to the rest of my article, Data Integration: Beyond ETL and Data Warehousing, on the Information Management website.

Business Intelligence (BI) Index: Weekly Update 06-26-2009

The Business Intelligence (BI) index continues to outpace the broader market indexes with YTD gains of 28.4%. The attached chart (click on the image to see a larger version) compares these indexes with broader market indexes along with a select set of high tech ETFs. High tech, as we has discussed, is currently the industry leader for the year and is likely to provide leadership with an upward market.

 
 

People, Process & Politics: Business & Integration

In our previous post People, Process & Politics: Stop the (Integration) Madness we discussed how we have promoted integration silos by  fragmenting those efforts across organizations, applications and projects.

Ending the integration silo cycle requires a fundamental change - viewing data integration holistically. From an architecture level, we need to view all the pieces: information, data, technology and product. More importantly, we need to view data integration from political and organizational perspectives. Most IT projects fail due to people problems, not because of technology.

To establish data integration as a business asset, you need to create the following:

• Business and IT case for viewing data integration as a fundamental business problem that needs to be addressed
• Integration investment portfolio with data integration as an infrastructure project across all business and IT groups
• Integration Competency Center (ICC)

BI & On-Demand Indexes: Tech on a Run

Our business intelligence (BI) and on-demand software stock indexes have been on a strong run this year. As discussed, this has accompanied a overall market bounce from March lows and support high tech in its current leadership role in year-to-date performance.

MarketWatch in its market snapshot yesterday "Technology the brightest, most volatile star of stock market" commented on the tech sector. The article substantiated our earlier observations that tech is often an early stage leader but that the market, and tech, may have gotten ahead of itself. Caution in the near term is advised in this sector but long-term growth should be strong as we emerge from this recession.

The following chart listing S&P industry sector performance yesterday and YTD. The chart is derived from statistics from my favorite show to excercise to --  "Fast Money." The numbers were shown on the show and in "How Deep Will The Correction Get?"

 

BI & On-Demand Indexes: Weekly Update 6/19/2009

The Business Intelligence (BI) and On-Demand Software indexes continue to outpace the broader market indexes with YTD returns of 27.8% and 35.3% respectively as of June 19. The attached chart (click on the chart to see a larger version) compares these indexes with broader market indexes along with a select set of high tech ETFs that have also outpaced.

People, Process & Politics: Stop the (Integration) Madness

In previous post “People, Process and Politics - We All Hate Data Silos, So Why do They Happen?” we discussed why we have set ourselves up to continually repeat history and build data silos. Like an episode of The Twilight Zone we seem to be hopelessly locked into an endless loop.

Enterprises have fragmented their integration efforts across groups, applications and technologies. The classic split has been between the groups developing data warehouse and business intelligence applications versus the ERP (enterprise resource application) applications. Typically each group uses different technologies: ETL (extract, transform and load) for DW and applications using either data virtualization (formerly EAI Enterprise Application Integration) or EII (Enterprise Information Integration) used. Add in SOA (service oriented architecture) initiatives for application integration and an enterprise integration efforts are spread across multiple projects with different sponsors, funding, resources and objectives that often overlap.

If that is not a big enough problem, our industry has assaulted enterprises with a barrage of “emerging” technologies that are supposed to solve various integration problems. These solutions include Corporate Performance Management (CPM), Master Data Management (MDM), Customer Data Integration (CDI), Product Information Management (PIM), Enterprise Information Management (EIM) and many others. Each of these solves problems in specific domains but at their core they involve integration. Vendors selling solutions bundle integration technologies (the same as described above) with their solutions. The result is integration silos that create new data silos.

Admit It, You've Got a Problem

You must admit you have a problem before you can solve it. Many enterprises are blind to their integration silos. All they see is their investments in ERP, DW, BI, CPM, MDM, CDI, SOA and PIM applications and the resulting databases with terabytes of data stored in them. Smug with the knowledge that they have all the data that the business needs, they're not even aware of the data silos surrounding them created by their integration silos.

At some point, they start noticing inconsistent data, which is a symptom of an integration silo problem. However, because they don't understand the cause, they focus on relieving the symptoms with quick-hit solutions. For example, they may try to consolidate business intelligent (BI) tools. While this is a worthy goal unto itself, using a single BI tool will do nothing about the fact that the underlying data is coming from many disparate systems ERP, CRM, SCM, data warehouses, data marts) and is inconsistent. The business is not getting different numbers because it is using different BI tools, but rather because each tool is associated with a different database where the data had been transformed differently than the other databases. The BI tool used is the tip of the iceberg; the data integration issue is what is below the waterline.

Once you recognize an integration silo problem, you can think out-of-the-box on how to define, approach and attack it.

Next post on “People, Process & Politics” we will begin to discuss how to stop the integration madness.

High Tech: Leader of the Pack

The Dow Jones Index (DJIA) moved into positive territory for the year on Friday, June 12 with the S&P 500 at 4.79% year-to-date (YTD). Although much has been written about the broader indexes bouncing off their lows and the state of the various industries being bailed out, it has been the tech sector that has been one of the leading industry groups this year from a stock performance perspective. The NASDAQ 17.87% YTD gains leaves the broader indexes in the dust.

There are several reasons for technology sector’s performance.  First, tech companies are generally in better financial shape than many other industries.  Many tech companies, especially those in business intelligence and data warehousing, have little if any debt and are therefore not directly impacted by the credit crunch (of course their customers are so that impacts sales.)  Software companies are often cash flow positive even in lean times with maintenance and service revenue offsetting slowing new license sales. The iShares S&P North American Technology-Software Index Fund, is up 24.3% YTD illustrating investors fail in software potential.