Data Warehouse Considerations: When and Why?

A little background here:

I know what a data warehouse is, more or less. I've read several dozen guides on data warehousing, I've played with SSAS, I know what a star schema and a dimension table and a fact table is, I know what ETL is and how to do it. This is not a "how" question or a request for tutorials.

My issue is that all of the material I've read on data warehousing seems to gloss over the rationale for building a data warehouse. They all figuratively, or in some cases literally start with the phrase "so you've decided to build a data warehouse..." Except I haven't made that decision yet.

So I'm hoping that SO members can point me to, or help come up with, some kind of semi-objective test. Something that I can adapt to a particular system and end up with either "yep, we need a data warehouse" or "no, the payoff today would be too small." I think that the specific questions I should be able to answer are:

1. At what point is building a data warehouse an option worth considering? In other words, what telltale signs, metrics, or other criteria should I be looking out for that might indicate that a standard transactional environment is no longer sufficient?

2. What are the alternatives to a full-on data warehouse? Denormalization in the transactional database and the bog-standard replicated "report server" are two that come to mind; are there any others I should explore before committing to the DW?

3. Why is a data warehouse better than said alternatives? If the answer is, "it depends", then what does it depend on?

4. When shouldn't I attempt to build a data warehouse? I'm skeptical of anything declared as a "best practice" irrespective of context. Surely there must be some scenarios where a DW is the wrong choice - what are they?

5. Are there any practical examples I could look at of systems that were improved by introducing a data warehouse? Something that would explain to me, end-to-end, what sorts of decisions or analysis they needed the warehouse for, how they decided what to put in it, and how the warehouse ended up fitting into the larger environment? I don't want a contrived "let's make a cube out of the AdventureWorks database" - the implementation is irrelevant to me, I'm interested in the specifications and designs and overall thought process that were involved.

I generally try not to ask multi-parters but I think that these are all very closely-related. I'm willing to accept any answer that addresses at least the first 4 questions, although the last would really help to crystallize this in my mind. Links are fine if somebody's already written about this, as long as they're reasonably concise and specific (link to Ralph Kimball's home page = not helpful).

Hope I've made the question clear - thanks in advance for your answers!

I'll see if I can do my best to answer your questions succinctly.

> 1.At what point is building a data warehouse an option worth considering? > In other words, what telltale signs, > metrics, or other criteria should I be > looking out for that might indicate > that a standard transactional > environment is no longer sufficient?

a. If you find that reporting and monitoring are impairing the performance of your production system and/or an offline data store.

b. If you find that getting answers to your business questions requires building a lot of complex SQL each time.

c. If you find that every time you make a change to your transactional schema, you have to go back and rework all of your reporting queries.

d. If you want to bring together data from multiple sources.

> 2.What are the alternatives to a full-on data warehouse? > Denormalization in the transactional > database and the bog-standard > replicated "report server" are two > that come to mind; are there any > others I should explore before > committing to the DW? > > 3.Why is a data warehouse better than said alternatives? If the answer is, > "it depends", then what does it depend > on?

I'll answer these together. I wouldn't think of a data warehouse as an all or nothing venture. It's simply a concise phrase that means "storing your data in a way that allows you to more easily and quickly answer business questions."

Transactional databases are designed to efficiently interface with applications. Data warehouses, data marts, operational data stores and reporting tables are built to efficiently interface with people, if that makes sense.

> 4.When shouldn't I attempt to build a data warehouse? I'm skeptical of > anything declared as a "best practice" > irrespective of context. Surely there > must be some scenarios where a DW is > the wrong choice - what are they?

Good question. If your transactional system provides you with sufficient insight into your business, you probably do not have a need for warehousing.

If you only have one source of data and performance is not a problem, you can probably gain insight from creation of simple reporting tables.

> 5.Are there any practical examples I could look at of systems that were > improved by introducing a data > warehouse? Something that would > explain to me, end-to-end, what sorts > of decisions or analysis they needed > the warehouse for, how they decided > what to put in it, and how the > warehouse ended up fitting into the > larger environment? I don't want a > contrived "let's make a cube out of > the AdventureWorks database" - the > implementation is irrelevant to me, > I'm interested in the specifications > and designs and overall thought > process that were involved.

That's a big question that would take far more space than I'm allotted here.

On this one, I can point you to a few places that might provide the insight you seek.

• "Implementing A Data Warehouse: A Methodology that worked" by Bruce Ullrey is a book documenting one man's journey to building a data warehouse. It's not highly polished, which gives it more realism. It reads like a journal with lots of models and other visuals that illustrate his efforts pretty well.
• "Business Intelligence Roadmap" by Larissa Moss. Standard fare. Walks you through the process of building a BI practice at a high level.
• "The Profit Impact of Business Intelligence" by Steve Williams gives a number of case studies that show the value of building data warehouses.

1. The main purpose of a DW is to speed-up (simplify) reporting and analytic. It enables slicing and dicing of data in any way a business user can think of.

2. For a first step DW, you can simply implement a Kimball star schema and run SQL queries against it. If this proves to be still too slow, start thinking about pre-calculated aggregations (cubes).

3. The slicing and dicing of information against a DW is way simpler, than against a normalized DB. Replicated report server will improve performance, but will not simplify slicing and dicing. Also keep in mind that the DW belongs to business users, so it is up to them to come up with various slice/dice ideas at any time -- IT people should simply provide environment in which something like this is possible.

4. If you just run few reports from time-to-time on your operational system and are satisfied with performance, there is no need for DW.

5. All my experience is with systems where business users endlessly complain about slow reports and inability to write "complicated queries", while production people complain that the database gets bogged down due to reporting. In all cases a simple Kimball star and a report server with cache and snapshots were good enough.

1. You should consider building a data warehouse, when two of the following criteria match:

• Huge amount of data
• Many big complex selects (possibly compared to few inserts, updates, and deletes) that just take too long to execute (and are complicated to write)
• Data from different systems needs to get combined

2. It's really the question what you consider a data warehouse. In many cases you can move gradually from OLTPs Systems with some reports to a full blown data warehouse, as long as you can stick to a relational database management system. First could be to build a first fact table, and keep using the normalized tables for dimension. Then adding more facts, more fact tables or dedicated dimension tables to the game. First in the same database (or one of the databases of the involved systems), possibly moving to a separate database later.

3. A full data warehouse (separate database, star schema) offers the best options for tuning select statements, apart from going to a specialized system. It is also cleanly decoupled from the OLTP system(s). Think schema design, but also resources like CPU, I/O and memory and organizational, like scheduling of new releases. Of course it is a lot of work which you possibly don't need.

4. It's in the answers above: just because you have a handfull of complex queries, doesn't mean you should build a DWH, same holds for the other criteria, if they come in isolation.

5. Can't offer much here, but the advice: go agile. The requirements for a DWH depend extremly on the possibilities the users see. There for requirements are likely to change. Automating tests with databases is a pain, but fooling around in a production system with no proper tests is worse.

> At what point is building a data warehouse an option worth considering? In other words, what telltale signs, metrics, or other criteria should I be looking out for that might indicate that a standard transactional environment is no longer sufficient?

I'd recommend a data warehouse when you observed that performing reporting and analysis activities on the in the transactional data store was harmful to both.

> What are the alternatives to a full-on data warehouse? Denormalization in the transactional database and the bog-standard replicated "report server" are two that come to mind; are there any others I should explore before committing to the DW?

I have nothing to offer here. I'd say that keeping the transactional and reporting databases seems sensible to me, regardless of whether you call it a warehouse or not. Data mining can be a very CPU intensive activity.

> Why is a data warehouse better than said alternatives? If the answer is, "it depends", then what does it depend on?

I have nothing to offer here.

> When shouldn't I attempt to build a data warehouse? I'm skeptical of anything declared as a "best practice" irrespective of context. Surely there must be some scenarios where a DW is the wrong choice - what are they?

I'd say that if you don't need to keep long history, aren't doing intensive analysis of the data, and your reporting needs are limited to an ad hoc query from time to time, then perhaps a data warehouse isn't necessary.

> Are there any practical examples I could look at of systems that were improved by introducing a data warehouse? Something that would explain to me, end-to-end, what sorts of decisions or analysis they needed the warehouse for, how they decided what to put in it, and how the warehouse ended up fitting into the larger environment? I don't want a contrived "let's make a cube out of the AdventureWorks database" - the implementation is irrelevant to me, I'm interested in the specifications and designs and overall thought process that were involved.

My employers have all used data warehouses for many years prior to my arrival, so I can't speak to what things were like before I arrived.

From my experience, the first sign for starting to think about data warehousing is when you have (or are developing) a transactional database and the users start adding lots of reporting and data history requirements. Which is pretty much always. It's always easier to have a separate data warehouse or reporting database than to try to design a transactional system that handles the reporting needs that end users always have. Storing history (for business entities) in a transactional system adds complexity and bloats a database that should be as responsive as possible.

On the flip side, I've been in large companies where many groups created data warehouses because data of interest was spread across many systems and was therefore difficult to query. The problem was that each group created their own data warehouse because all the existing warehouses in the company did not have the right subset of information, or had a data model that was regarded as non-optimal or incorrect. This made the situation worse by creating even more disparate data systems that were hard to compare.

DW could be considered if, one is using a ‘Transactional System’ from a long period. Later, they realize that they need to perform some data mining, to determine different data patterns of the business. And finally, with the help of the determined data patterns, one wants to help the top management to take further decisions in the benefit of the company.

Following steps needs to be taken up for building up a data ware house:

1. An ETL platform and database needs to be decided for the database.
2. A reporting tool like SSRS, Tableau, etc. needs to be chosen for the visualization.
3. One may opt for the Data Analytical language like R, for further use.
4. Finally, all this will help in developing the data ware house and reporting tool. 

"I think that why do some projects fail?"

There are five primary reasons:

• lack of partnership between the IT department and business users;
• incorrect data warehouse architecture;
• not enough experienced people;
• improper planning, such as failure to use a proven methodology and a plan to ensure that no details are omitted;
• and depending on bleeding-edge technology.