One thing I appreciated in the article was its realistic tone, which recognized the existence of shadow systems and the role that they play in the real world. While the article focused on spreadsheets, this doesn’t mean that they are bad and that centralized CRM is good (as a matter of fact I might argue the opposite in many cases). The real issue being discussed is the risk that occurs when information architecture doesn’t match the needs of the organization.

Looking Globally

For many low-importance, one-off operations, a spreadsheets is fine. It can be shared easily, and usually owned by one person. However, what normally happens is that someone sharp (and usually not someone in IT) decides to do try to build some infrastructure around a spreadsheet without considering the consequences. It is quick, cheap, and easy to do, to a point. The problem is that the point when a spreadsheet becomes unmanageable is often well after the point where an organization depends on its output to function effectively.

For any system, an information architect needs to consider:

• Cost
• Effectiveness
• Visibility
• Traceability
• Mangageability
• Quality
• Auditability

Furthermore, the situation needs to be considered from a forward-thinking perspective. That is, we need to try to understand how the landscape is going to look a few years down the road, and to make our systems flexible enough so that we don’t put our organization into a bind if we are wrong.

Spreadsheets are normally built only considering cost and effectiveness, something that is only discovered after time has gone on. It is often the case that a system would be much more effective living in a database, BI tool, or custom software application when considered over time.

He writes:

Authorative sources of data are indeed useful - just don’t count on them to tell the truth, the whole truth and nothing but the truth.

I believe that one of the biggest problems that we have in dealing with data is the false belief that for every organization and situation, there is a single view of information that can satisfy everyone’s needs. Now, this isn’t a technology problem and it isn’t a data problem, it’s an organizational problem.

The Myth of the Single View

In any organization, we end up with different groups with different needs, normally based around:

• Speed
• Reliability
• Accuracy
• Cost

Each group has specific needs based on their own situation. For example, when looking at customer data, the people in HQ might not care if every customer account has the most up-to-date address available, but the people in the warehouse certainly do. At the same time, the people in the warehouse don’t care about how much it costs to , while the people in HQ are much more focused on the bottom line.

Get these folks together in a room and you will have a terrific argument about what the organization needs and and how it is going to be done (BTW, there is a related post to this on the wonderful Creating Passionate Users).

While this sounds like a problem for human resources or general management, this phenomenon is usually expressed as a function of IT, because that is where the rubber hits the road. Since IT is often a shared resource and has a vested interest in interoperability, the issues of culture and organization come out as a function of architecture development.

An Honest Assessment

The honest truth is that there isn’t a single view of the business, its data, or its processes, that is going to meet the needs of the entire organization. A lot of vendors and consultants for CRM and MDM solutions are going to try to tell you otherwise, realize that they are selling something as they do this. The answer is that this is a complicated world, and things aren’t getting any easier.

If your IT is going to represent the entire organization, you must embrace complexity and understand the fact that there are going to be a cacophony of voices and a host of diverse world views that all exist simultaneously and are all using and competing for the same resources.

In years past, metadata was the domain of data architects. It helped them understand what data they had and how it related to the sources and operations from which it came and to which it went. At the first mention of metadata business users would roll their eyes and head for the conference room door. Surely metadata was the stuff of arcane IT discussions best had out of earshot of people driving and running the business.

Then metadata management progressed and someone had the silly idea of articulating the business value, the value to the business side of the house, for metadata. The value came from the resolution of an age old problem. A corporate manager is sitting in a conference room looking at their regular monthly sales report and it is different from what they expected based on anecdotal evidence from the field: the numbers are too low.

Personally, I think that this recent interest is driven by a few things:

1. Regulation and the threat of real penalties for inaccuracies in reporting. People got interested enough to protect their own hides.
2. The rise of ERP and BPM in the marketplace. If everything is in one place then metadata suddenly becomes a lot easier to manage.

Truthfully, I wonder how all of this is going to turn out. I know there are lots of people who want to sell metadata software, but in my experience it takes a lot of resource (time, effort, and expertise) to maintain a comprehensive metadata environment. The threat of jail time helps to keep people motivated enough to save their necks, but not enough to make something useful. Being locked into an ERP package can mean the same thing, only it is your data that is locked.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

• 
• 
• 
• 
• 
• 

]]> http://www.architected.info/blog/musings-on-metadata-and-compliance/feed/ http://www.architected.info/blog/tales-of-information-quality-are-greatly-exaggerated http://www.architected.info/blog/tales-of-information-quality-are-greatly-exaggerated#comments Mon, 25 Sep 2006 17:41:33 +0000 morgan Uncategorized Information Quality In the News http://www.architected.info/blog/tales-of-information-quality-are-greatly-exaggerated The Scotsman wrote about a woman who returned home from the hospital to find a letter from the water company notifying her of her own death. While my guess is that this was a problem with information quality at the water, the breakdown could have happened at the hospital, in the government, or somewhere inbetween.  Regardless, it certainly is a black eye for data stewardship for someone.
Ignoring the recursive metaphysical implications of this situation, this really hits home how important information quality is in every organization.

NOTE: I originally spotted this on the upcoming page on Digg.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

• 
• 
• 
• 
• 
• 

]]> http://www.architected.info/blog/tales-of-information-quality-are-greatly-exaggerated/feed/ http://www.architected.info/blog/working-on-borrowed-productivity http://www.architected.info/blog/working-on-borrowed-productivity#comments Fri, 08 Sep 2006 14:13:20 +0000 morgan Databases ETL Information Quality Practices Transformation Reporting http://www.architected.info/blog/working-on-borrowed-productivity Project X Discussions has an interesting article about project staffing and the bloat that often occurs with data related projects.

In a number of client environments I have often been amazed by the number of people that can be assigned to a project. Project Managers, Business Users, Business Analysts, Architects, Technical Analysts, Developers, Database Analysts, Database Administrators, Data Modelers, Subject Matter Experts, Testing specialists, Data Assurance people, Production and Operations People and of course a couple of people like me, the consultants.

While it is impossible for one person to do everything I often wonder how many people on such a project team could be removed from the project without impacting and perhaps improving the outcome.

They make some good points. In a structured environment it is very easy to get project bloat due to specialization and role playing. In my experience, a few smart, determined generalists deliver more than a pack of highly qualified specialists most of the time.

The Twist

The article then goes on:

One person I worked with recently proved that a single person acting alone can accomplish amazing things if they have access to all the right tools and know how to use them. In this scenario the end user need some reports - in excel format.

My friend, used UNIX scripts against flat files and SQL queries against the database to create a number of SAS data sets. He ran SAS functions against the data to aggregate it and exported it into Excel for the end user. It took him a couple of days.

It is easy to confuse project bloat with the need for high-quality business processes, and we have to be careful in this regards. I believe that this person was working with what I call “borrowed productivity”. He was able to deliver his part quickly, in large part by doing things that will make things more difficult (and expensive) in the future. He avoided all the processes that will save money over time with a quick fix immediately.

Where the organization will pay down the line will be:

1. Quality.
2. Sustainability.
3. Reusability.
4. Consistency.

My Experience

I once worked for a large (Fortune 500) company who had a division that did almost all of their reporting from Microsoft Access databases. They were able to crank out report after report and get them into excel format and delivered via email very quickly. At one point, this group was generating millions of emails

At the same time, there were a lot of things that didn’t work on this model:

• The reporting group was really never able to get their reports to match up internally, not to mention reports generated through the centralized data warehouse.
• When business processes or changed it was very difficult to find all the places where code needed to be updated.
• There were a huge number of reports that were very similar, with only slightly different parameters.
• When more reports were needed the only recourse was to hire more Analysts.
• The users soon grew tired of having a large number of excel spreadhseets in their Inbox each morning and stopped reading many reports

Eventually, this reporting system became unsustainable and the group went through several major crises and ended up being mostly disbanded. This was a huge waste of resources and a bitter loss of business expertise and technical talent. A lot of good people lost their jobs because of a bad system.

Conclusion
Small teams are good, very good. They are efficient and often much easier to run. Small thinking is bad, very bad. It often unintentionally deceptive and very expensive in the long run. There is no causation between big groups and small thinking or small groups and big thinking, but often there is a correlation.

Regardless of your environment, in developing your inormation architecture make sure that you understand that:

1. Data delivery has to be fast, at least as fast as the business that drives it.
2. Initial development is usually the smallest cost in the data lifecycle.

Normally, a discussion around this subject will neglect one side or the other, but this really is just a disservice to the efficiency of your organization.

Other Reading

Rick Sherman has discussed shadow systems , this is one of the few works I have seen that view the discussion holistically. I would highly recommend taking a look at his work.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

• 
• 
• 
• 
• 
• 

]]> http://www.architected.info/blog/working-on-borrowed-productivity/feed/