Among the technology projects that companies are pursuing today, one of the most risky, expensive, and failure-prone among them is building a data warehouse. While data warehouses can deliver enormous benefits, creating and maintaining one on your own is fraught with difficulties that can easily undermine your efforts. In this article, I will explain why this is the case. I will also offer a way to implement a data warehouse that can deliver greater benefits than a homegrown warehouse with considerably less risk.
Before I say more on this subject, I want to make one thing clear. After deploying hundreds of business intelligence (BI) and reporting systems at JD Edwards sites, Andrews Consulting Group knows that it makes much more sense for such systems to access data warehouses instead of production databases. Here are just a few of the problems that companies encounter when they use BI tools to access production databases.
- The BI tools must contend with transaction processing systems for production databases. Since transaction processing takes priority, BI performance inevitably suffers.
- New releases of the production database or ERP applications introduce changes that make it necessary to rewrite reports every time an upgrade occurs.
- Field names in production databases are hard to decipher and are often stored in odd formats. This makes it difficult for BI users to work with the data.
- There is no easy way to integrate information from sources besides the production database into reports and analyses.
For these and other reasons, it makes sense to run your BI and reporting tools on a data warehouse that is dedicated to BI and reporting functions. That said, building your own warehouse could be hazardous to your corporate health. Over the years, our team has been approached by dozens of JD Edwards customers seeking help on failed data warehouse projects. In some cases, these warehouses were built from scratch. In others, they were built from templates that were provided by a software vendor or consulting firm, then customized by our clients. In either case, we have seen large percentages of these projects fail completely or only deliver a fraction of their intended benefits.
Why Do “Build Your Own” Data Warehouses Fail?
There’s a simple reason why so many homegrown data warehouses yield underwhelming results. When an organization decides to build and maintain its own warehouse, it is really deciding (whether it knows it or not) to get into the software business. Like a software vendor, it must assume responsibility for modifying the warehouse when any of the following events take place:
- The company upgrades to a new release of its ERP software, adds new modules, or modifies the existing ones
- Users demand that new data or functionality be added to the warehouse
- The company deploys a new release of its existing database or just patches the current release
- The company switches database platforms or moves to one of the data warehouse appliances
- Updates or changes must be made to the extract/transform/load (ETL) tools that move information from the production database to the data warehouse
Any of these situations can trigger significant changes to a data warehouse. Moreover, since one or more of these events is almost always taking place within companies, the flow of changes can be constant. That is why most warehouses require a dedicated team with considerable skills in databases, ETL tools, and BI to meet new demands and maintain compatibility with other software. Since most companies cannot recruit or adequately fund such teams, their homegrown warehouses eventually sink under rising tides of unmet requirements.
Does your company really want to be in the BI software business? Before you answer this question, consider the following. As a JD Edwards customer, you don’t build your own ERP applications. You buy a package that someone else (Oracle) supports. So if you could find a vendor-provided data warehouse that meets the majority of your requirements out of the box, is flexible and extensible, and is kept current with all the software you use, why not use it instead of building your own?
The Low-Risk Alternative
I bet that many of you are saying to yourselves right now, “That’s all fine and good, but there are no such data warehouse products.” That was true for many years, which is why most people still think that data warehouses must largely be built from scratch. Recently, however, BI and reporting vendors have begun to realize that if they want to sell data warehouses to mid-size companies, they must “productize” them like ERP vendors have with their offerings. As a result, a new generation of data warehouse packages has entered the market. These packages provide the robust functionality and support levels that ERP systems have offered for years.
Among those packages, one called RapidDecision is designed specifically for JD Edwards EnterpriseOne (as well as its predecessor OneWorld) and World users. RapidDecision provides a comprehensive data warehouse and pre-built data marts that work with all of the databases on which JD Edwards products run. It also supports many of the popular ETL, reporting, and BI tools. Whenever Oracle enhances EnterpriseOne or World, or when ETL and BI tool vendors produce new releases of their products, RapidDecision is updated to support the new releases. In addition, RapidDecision can incorporate data from other Oracle and non-Oracle applications.
One advantage of RapidDecision that should be noted here is that it is built to have the minimum definable impact on JD Edwards software, its database and hardware, the network it runs on, and the BI tools that use it. I know that is a lot to say, so let me explain. Most data warehouses use a “refresh approach” that simply copies data on a timed repeated basis from the production database to the data warehouse. Before each refresh, you are usually required to bring down your BI tools and the data warehouse. You must then run a massive query on the JD Edwards system, move the resulting data to the data warehouse, and then transform the data.
While this approach is the norm for many data warehouses, it is highly intrusive and disruptive to both the JD Edwards and BI systems. Hopefully, you will not be the person telling the accountants that they have no access to the system because of a three-hour refresh. In addition, refreshes are wasteful because they move 99.9% of the same data as yesterday to get the 0.1% that has been added or changed.
By contrast, RapidDecision only moves the adds, changes and deletes as they occur. It does this without touching a single piece of JD Edwards data in the tables or by changing the JD Edwards table structure. It simply reads the native database logs to retrieve the added, changed or deleted data. This is what enables it to have the minimum impact on system performance and availability.
With RapidDecision, companies with limited IT resources can outsource the deployment and maintenance of a robust data warehouse, freeing their own IT staffs to do what they do best. As the chief architect of RapidDecision, I can tell you that we have assembled a top-notch team of developers to keep our offering current with all the products that JD Edwards customers use. We have also refined the deployment process for RapidDecision so that we can get the product up and running in as little as five days. In short, we make it possible for companies to deploy a “best practices” data warehouse along with all the necessary BI tools, but leave the risks and support headaches with us.
If you would like to learn more about RapidDecision, there are several things you can do. First, you can visit the RapidDecision web site. Second, you can watch an online demonstration of RapidDecision’s functionality (done by yours truly). You can also watch a helpful (and humorous) demonstration of how a data warehouse works. Finally, you can attend one of our free monthly webinars about the product. You can also send us an email if you have specific questions you would like to ask or pose your questions in the comment box below.
Whatever you choose to do, I hope that you take the time to consider RapidDecision before building your own data warehouse. It may just transform your thinking about what a packaged BI solution could offer your organization.