« Discount on my Online DW and BI Data Modeling Course | Main | Data Integration: Hand-coding Using ETL Tools Part 2 »

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8345444f069e20120a6537656970b

Listed below are links to weblogs that reference Data Integration: Hand-coding Using ETL Tools:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Justin

I've seen eaxctly those problems. I then came across Talend first as an open source tool but then we decieded to pay for the additional functionality in the commercial version - its as good as the others in Gartners magic quadrant but at a fraction of the cost. Take a look and you won't be disappointed.

Rick Sherman

Justin,

Talend is an attractive open source ETL option. I haven't used it at clients but have used it in workshops when I teach at an engineering university's Master Degree program. It wouldn't win a functionality bake-off with the upper quadrant tools BUT not every integration project needs all those functions anyways. It may not be "best" overall but it may be "best fit" and the price is right.

Too many people though decide that open source is free and do not budget any time to learn wither the tool or the fundamentals & best practices for ETL (or data integration.)

Thanks for your good observations,
rick

Michael Waclawiczek

Rick, there shouldn't be anyone disagreeing with your argument against hand-coding these days as there is a growing number of very capable ETL tools on the market that are either free or highly affordable. I'd like to point out, however, that you should have included expressor in your analysis of classes of ETL tools that are capable of handling many, if not all departmental and SMB needs. Although we are not a “free” tool, we are very much targeting departmental and SMB opportunities because there are many accounts that need more power and functionality than what the freeware or open source software tools can offer.

For example, we find that a sizable percentage of Microsoft SSIS users (30 - 40% in our estimate) are running into performance and/or data transformation complexity and/or data connectivity issues. Of course, the Gartner upper-quadrant ETL tools could solve these problems, but we believe expressor is the only solution on the market that's both affordable and matches the upper magic quadrant performance, data transformation complexity, and connectivity capabilities of the likes of Informatica, DataStage, and Ab Initio.

A case in point, one of our new customers is American Tower, who is replacing SSIS with expressor for exactly the reasons I mentioned earlier. Last week we also exhibited at the PASS Summit 2009 and got a very warm reception from many SSIS users who are facing a similar dilemma, which they often expressed as “Do I continue with SSIS because it's free? I can't afford the 250,000 dollars or more price tag of any of the upper quadrant tools.” And as you pointed out, none of them wants to go back to hand coding. Which explains why they were so enthusiastic about expressor – the high-performance ETL alternative that's up to 80% less expensive than the traditional high-end ETL vendors.

Rick Sherman

Michael,

Thanks for the feedback and I agree that expressor is a great option people should consider.

The "bundled" tools, especially SSIS, are better than hand-coding but often run into limitations. They often come with a limited set of pre-built functionality (the developers have to augment the gaps with SQL code), performance is highly dependent on database tuning and they often do not scale.

I'll keep spreading the word that IT should embrace ETL tools and learn data integration processes. I'll also try to position ETL tools in respect to what they can do, their ease of use (to learn and to develop with), their TCO (total cost of ownership)and their fit for the task & organization at hand.

Thanks,
Rick

The comments to this entry are closed.