Data Doghouse - performance management, business intelligence, and data warehousing

'Big data' analytics programs require tech savvy, business know-how

Articles and case studies written about “big data” analytics programs proclaim the successes of organizations, typically with a focus on the technologies used to build the systems. Product information is useful, of course, but it’s also critical for enterprises embarking on these projects to be sure they have people with the right skills and experience in place to successfully leverage big data analytics tools.

Knowledge of new or emerging big data technologies, such as Hadoop, is often required, especially if a project involves unstructured or semi-structured data. But even in such cases, Hadoop know-how is only a small portion of the overall staffing requirements; skills are also needed in areas such as business intelligence (BI), data integration and data warehousing. Business and industry expertise is another must-have for a successful big data analytics program, and many organizations also require advanced analytics skills, such as predictive analytics and data mining capabilities.

>>>continue reading Big data' analytics programs require tech savvy, business know-how

The wrong way: Worst practices in 'big data' analytics programs

“Big data” analytics is hot. Read any IT publication or website and you’ll see business intelligence (BI) vendors and their systems integration partners pitching products and services to help organizations implement and manage big data analytics systems. The ads and the big data analytics press releases and case studies that vendors are rushing out might make you think it’s easy -- that all you need for a successful deployment is a particular technology.

If only it were that simple. While BI vendors are happy to tell you about their customers who are successfully leveraging big data for analytics uses, they’re not so quick to discuss those who have failed. There are many potential reasons why big data analytics projects fall short of their goals and expectations. You can find lots of advice on big data analytics best practices; below are some worst practices for big data analytics programs so you know what to avoid.

>>>continue reading The wrong way: Worst practices in 'big data' analytics programs

BI training too important to skip

Lauren Gibbons Paul interviewed Howard Dresner and me for an article on training, which was published in SearchManufacturing ERP.com this week:

Training is the final step of a software implementation and the one everyone loves to skip. The scenario is all too common. Companies spend time up front justifying the purchase, selecting the software and implementing it and then decide they’re done -- giving short shrift to the crucial last piece.

When it comes to business intelligence (BI) software implementation, this mistake could render the whole undertaking null and void. BI training that goes beyond the tool’s technical features is crucial to helping employees gain the insights they need to do their jobs better, industry observers say.

>>>read the rest of this article

Speaking of BI training, some things never change. The situation is not so different from years back when I wrote this article: Training? We don't need no stinkin' training ...

BI’s Dirty Secrets – The Unfortunate Domination of Manually-Coded Extracts

Manually-coded extracts are another dirty secret of the BI world. I’ve been seeing them for years, in both large and small companies. They grow haphazardly and are never documented, which practically guarantees that they will become an IT nightmare.

How have manually-coded extracts become so prevalent? It’s not as if there aren’t enough data integration tools around, including ETL tools. Even large enterprises that use the correct tools to load their enterprise data warehouses will often resort to manually-coded extracts to load their downstream BI data sources such as data marts, OLAP cubes, reporting databases and spreadsheets.

After seeing this problem in enough client companies, I’ve got a few theories as to why it happens:

• Money: The top tier tools are expensive. They are out of reach for SMBs and can even be too expensive for large enterprises to expand their use from the EDW to BI data source. There are data integration tools that would do a great job spanning price ranges, but for the most part nobody knows about them. And when they are used, they are misused (see below), so their reputation for producing a solid business ROI is diminished.
• Stretched resources: In large enterprises, the centralized data warehouse team likely has data integration experience, but their backlog of work means that people creating BI data sources are on their own. So they end up hand-coding. In SMB firms, the IT staffs are too small to dedicate anyone to data integration, so no one is an expert.
• Data never sleeps: Regardless of the state of data integration expertise and investment at an enterprise, business people still have to run and manage the business. This requires data. If the data has not been integrated for them, they’ll  figure out some other way to get it -- even if it means cutting and pasting data from spreadsheet queries or getting IT to “crank out” SQL scripts. This is why data shadow systems or spreadmarts get started and then become so prevalent.
• You don’t know what you don’t know:  Even when enterprises use data integration or ETL tools, they often don’t use them well. The biggest reason why people misuse these tools is that they don’t have a firm grasp of the concepts of data integration processes and advanced dimensional modeling.  Tools are easy; concepts are harder.  Anyone can start coding; it’s a lot harder to actually architect and design. Tool vendors don’t help this situation when they promote tools that “solve world hunger” and limit training to the tool, not any concepts.  

So, here’s what happens:  instead of using data integration best practices, people design the ETL tool processes the same way they would create a sequential series of SQL scripts to integrate data.  In fact, many an ETL process simply executes stored procedures (SP) or SQL scripts. Why use the tool at all if you’re not going to use its, capabilities? When this happens, IT figures it was a waste of time to use the ETL tool to begin with, and the ETL tool investment had no ROI. This becomes a self-reinforcing loop enabling IT to justify (or rationalize) manual coding.

• Coding is easier than thinking:  There is an inherit bias for the IT staff to generate SQL code. They know it (just like the business person knows spreadsheets), they can crank something out quickly and it does not cost anything extra. The typical scenario is that the IT person creates a SQL script or a stored procedure to pull data from one source and things are fine. But then several hundred SQL scripts or stored procedures later, the hodgepodge and undocumented accumulation of pseudo ETL processes becomes the recurring method to load the data warehouse or BI data sources. Each change to that set of code takes longer and longer.  It consumes more and more resource time just to maintain it.  When new data needs to be integrated, another IT person starts the next hodgepodge of undocumented code with yet another simple SQL script.

How do we get out of this mess? Stay tuned for a future blog post.

Prepare for the Big-Data Labor Shortfall

Where are all the big-data people? A recent front-page article in the Boston Globe raised this very issue.

The article, “Mass. firms see riches, jobs in charting oceans of data”, discussed reports on Big Data by the Mass Technology Leadership Council (MTLC) and McKinsey Global Institute regarding the role Massachusetts plays as a hot bed for both Big Data software firms and the people with the skills to level Big Data in their businesses. 

Big Data’s promised business benefits range from capturing more of a consumer’s wallet to discovering the cure for cancer.  Software firms certainly see the potential of Big Data. They’re investing in better ways to capture, integrate, manage and analyze Big Data. Both established high tech firms and venture capitalists are making significant investments in it.

The Big-Data Skills Gap

The gap we are going encounter in this industry is the shortage of both the skilled people to develop the Big Data analytics models, often referred to as data scientists, and the data-savvy business people to take advantage of them in their business. The McKinsey report states “By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.”

Our industry has faced a shortage of skilled people in business intelligence, analytics and data integration that has kept business from effectively using the data they already have. With the onslaught of Big Data and the advanced skills it requires, we’re destined to fall even further behind.

Data Scientist Definition

If we are ever to fill that gap we need to broaden our definition of what a data scientist is. This title and the job descriptions that are posted for these jobs are too concentrated on programming and IT skills. In order to create Big Data analytical and predictive models the person does need to have programming skills, but more importantly, that person needs to understand statistical modeling. To develop the model, the person also needs to understand their business and industry. It also helps if they understand how to apply customer behavior and economic models. 

Although Big Data means “a lot of data,” the reality is, in many cases, that data is incomplete and inconsistent.  Developing analytical models means dealing with dirty data and gaps.  Many “numbers” people have trouble dealing with these conditions.

Data scientists need to have expertise in statistics, economics and business besides the basic programming skills. Although the ranks of the software engineers and IT is an excellent place to start, our industry needs to broaden its appeal and reach to people with a mathematics, actuarial, statistical, economics or business backgrounds if the data scientist gap is ever to be filled.

McKinsey mentions the shortage of the data-savvy business managers who will be the users of the Bid Data analytical models developed by the data scientists. Fortunately, many business schools have started offering analytical curriculum at both the undergraduate and graduate level. For business people with deep business expertise, it is time to brush up on statistics.