I read a couple of interesting articles this week that really helped to crystallize some thoughts I was having …

1. Slashdot had an article about the $10 Billion lost annually due to spreadsheet error and fraud.

2. Rick Sherman wrote an interesting article about “shadow systems” in this month’s DM Review.

He defines shadow systems as:

“… groups of spreadsheets and local, customized databases - often Microsoft Access and statistical databases - created by business groups to gather data for their users. While these systems provide exactly the information that business users are asking for, they are rarely part of [a] .. strategy.”

Shadow systems are an undeniably human response to very real, immediate needs. Something needs to get done, it needs to get done efficiently, and it needs to be done with the tools at hand. All too often, an organization will approach its IT group with a need, only to be rebuffed because it doesn’t fit easily into the existing architecture or there is a conflict around priorities. For example, a small department has a need that is critical to its operating function, but it cannot get the attention or the resources of IT, which is focused on meeting the needs of the overall business. In this case, a shadow system is the only logical way to meet the needs of the business needs in a timely manner.

A lot IT people would say that shadow systems are evil, a waste of resources, and should be stamped out of any organization. I would take issue with this characterization. Shadow systems are a legitimate strategy in their own right. However, the organization as a whole has to understand exactly what they are getting into. Let’s look at some of the true characteristics of a shadow system:

Pros

1. Low initial cost.
2. Very fast to deploy.
3. Extremely customized solutions.
4. Empowers customers and users of data.
5. Great way to prototype potential solutions to business needs.

Cons

1. Extreme (almost fatal) issues around quality and consistency ($10 billion a year!).
2. Duplicated effort across (or even within) organizations.
3. Difficulty in oversight and accountability.
4. Maintenance and upkeep cost are undefined and are often much larger than the implementation cost.
5. True cost is hidden from the organization at large.

Notice that almost all the “pros” look great to business users, who don’t really consider the “cons”. At the same time, all the “cons” send IT people screaming and running for the hills, completely ignoring most of the “pros”. These are the seeds of conflict within an organization. Personally, I think the most interesting problems to solve lie at the intersection of two different domains. Shadow systems lie exactly at the border between IT and the organization-at-large. Fun stuff!

A truly effective information architecture has to consider shadow systems as a legitimate part of the overall data solution. The need for a shadow system should be used as an opportunity for IT to partner with an organization to build something really useful. At the same time, shadow systems should be developed deliberately and not allowed to evolve (or devolve) without direction. I have developed a checklist of the necessary ingredients for a shadow system to be successful:

Morgan’s Laws of Shadow Systems

To be viable over the long-term, a shadow system MUST have …

1. Organizational sponsorship, including IT oversight and support.
2. A limited amount of pre-defined resources for personnel, hardware, software, and licensing.
3. A well-thought out strategy for risk-management, especially considering information quality and regulatory issues (SOX, HIPPA, etc.).
4. A complete lifecycle, including firm dates for deployment, integration with the overall architecture, and retirement.

technorati tags: shadow, systems, etl, iq, dq, data quality, spreadsheets

This entry was posted on Tuesday, June 6th, 2006 at 9:33 am and is filed under Systems Integration, Information Quality, In the News, People, Practices, Understanding. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.