Do I Need a Data Warehouse?

This is an obvious question, but should be taken seriously. For some businesses the answer is an easy yes, but for others, there may be better solutions. It is wise to seek professional advice when answering this question. Having an outside professional examine the question may cost some money up front, but they are far less costly early on in the development.

What specific Problems will it solve?

Write down the four, five, ten, whatever, number of problems that having a data warehouse will solve. The problems should be clearly stated, be very specific, and have testable criteria for success. Make sure you publicize these problems and get user and management feedback.

What are my available resources (time, money, and personnel)?

Knowing what your actual resources are at the beginning is critical for defining the development path. If your budget is low, consider down-scaling the effort. If time is short, consider using off-the-shelf products extensively. If your personnel resources are thin, consider outsourcing. Being realistic about your actual resources will help you prevent overruns and project disappointments. Promising a full-blown data warehouse when you only have resources for a database, will always doom a project. Also, don't be afraid to tell upper management that the resources are too small for solving the problem. Believe me, they would rather know up front than get a surprise during deployment.

What criteria will I use to measure success?

This is an often overlooked step in the problem definition. For every problem stated, you must define a means for determining the success of the solution. If you can't think of a success criteria, then the problem is not defined specifically enough. Stay away from problem statements such as "The data warehouse must hold all our accounting data." Restate the problem in quantifiable terms, like "The data warehouse must handle the current 20GB of accounting data including all metadata and replicated data with an expected 20% growth per year."

Should I outsource all, some, or none of the development and operation?

If you have in-house personnel that are under-utilized or have time to be assigned to the development process, then keeping most of the development in-house makes sense. If not, then you can either hire additional staff or outsource some or most of the development. I recommend that you do not outsource all of the development. You must have some in-house expertise available or at least strong upper management support. Otherwise you may end up with a very nice system that does not solve your problems. Strategic outsourcing makes sense in most medium to large data warehouse projects. The outsourcing contractor can supply the needed expertise and personnel at the various development phases. And when a particular phase is finished, you are not left with a staff member looking for something to do. You will probably find the up-front costs of an outsourcing firm to be higher than hiring in-house personnel. But the long-term savings will be far greater with a professional outsourcing firm than by retaining in-house personnel. Remember to make sure you feel comfortable with the outsourcer's style and abilities. You will working with them very closely. Don't just choose the largest or best-known source. How you and your outsourcer "mesh" is far more important than their list of clients.

Am I upgrading an existing system, converting from a legacy system, or developing from scratch?

Developing a data warehouse from scratch, as strange as it sounds, is by far the easiest. If you are in this situation, count your blessings. If not, upgrading existing databases or converting from one or more legacy systems will be your lot. Fortunately, you will have a long list of "things that don't work right" to begin with. Make sure that you fully understand what systems will still be in place after the migration and how they will be integrated into your intranet. If your budget is low, then consider using middleware and "web-like" products to layer on top of the existing system. With a more moderate budget, you can replace inefficient systems with newer and more powerful ones. Remember that computer hardware, memory, and disk space are cheap. It's the software and operations that are expensive. Powerful hardware can make even today's bloated software work faster. With a higher budget, consider migrating legacy databases to a single powerful commercial database. If you are not sure what the "latest and greatest" data warehousing products are, hire a professional data warehousing consultant. Their fee will be well worth it.

1. Clearly state the problem(s) you wish to solve.
2. Identify all data sources and formats.
3. Identify the users of the completed system.
4. Formulate a specific budget - time, money, personnel.
5. Ask identified users to specifically state what they expect the system to do.
6. Ask management to specifically state their success criteria
7. Separate their requirements from their "desirements." Only design to requirements. The enhancement phase is where you address the "desirements."
8. Group and "bubble-up" requirements.
9. Generate a prioritized requirements table listing the requirement, where it came from, the success criteria, and priority. Keep this table high-level. A table with a dozen requirements will be much easier to manage than one with hundreds.
10. Produce a detailed development schedule including hardware, software, personnel, documentation, and reviews. Include outsourcing requirements and long lead-time items.
11. Get a sign-off of the requirements, resource allocation, and schedule from top management before you go any further.

Note that items 5 & 6 will be asked throughout the development cycle since their responses will change when they see prototypes and when they are being trained. Be sure to update items 7 through 10 each time.

Beware of getting caught in the cycle of

You: What are your requirements?

Them: I don't know, what can you do?

Rapid Prototyping
The rapid prototyping method is illustrated in figure X. There are 5 keys to a successful rapid prototyping methodology:

1. Assemble a small, very bright team of database programmers, hardware technicians, designers, quality assurance technicians, documentation and decision support specialists, and a single manager.
2. Define and involve a small "focus group" consisting of users (both novice and experienced) and managers (both line and upper). These are the people who will provide the feedback necessary to drive the prototyping cycle. Listen to them.
3. Generate a user's manual and user interface first. You will be amazed at what you will find out by producing a user's manual first!
4. Use tools specifically designed for rapid prototyping. Stay away from C, C++, COBOL, SQL, etc. Instead use the visual development tools included with the database.
5. Remember a prototype is NOT the final application. Prototypes are meant to be copied into production models. Once the prototypes are successful, then begin the development processing using development tools, such as C, C++, Java, SQL, etc.