Folks,

I have removed the series Organizational Approaches to Data Quality, for a number of reasons.  When I started writing these articles I had one thing in mind, and as I wrote it seemed to turn into another.  Unfortunately, as I re-read the postings they don’t adequately describe any of the ideas I was trying to get across.  I will be re-writing a lot of this material and re-releasing it with more appropriate titles.

Thanks for your patience!

There was an interesting article in the Chicago Tribune about some information quality issues between the Minneapolis Police Department and the FBI. Since I had already discussed some issues around crime reporting in the UK, I thought I should try and balance things out across the pond

This story interested me at first because of one of its headlines:

The department said the FBI advised it to send the faulty figures to meet deadline.

Makes it sound like someone is being really pig-headed, doesn’t it? The usual story about the big organization vs. common sense. Originally, I wanted to call this posting “Quality vs. Deadlines” and get a chance to rant about how organizations prioritize information quality. That is, until I read the actual story.

Here’s what actually happened (in a nutshell):

1. The MPD discovered that there was a major issue in their process for gathering crime statistics.
2. The MPD notified the FBI that they did not want to submit this data until it was correct.
3. The FBI decided they would rather have bad data submitted so that the city could meet its deadline for its preliminary report (released this week).
4. The FBI will accept the corrected data and put it into the final report (released in October).

Sounds pretty reasonable, right? Everyone is trying to get the best information possible and incorporate it into the overall process. The FBI will put in the effort to make sure that things are right. The MPD is much chagrined about the error, didn’t kill the messenger, and took institutional ownership for the problem.

[A police spokesperson] was quick to compliment the crime analyst who noticed that the subcategory of aggravated assaults committed with a firearm wasn’t tabulated accurately, she said. “To his credit, he looks at hundreds of numbers everyday,” she said. “The Police Department isn’t exempt from technical glitches.”

And yet, there is a story in the newspaper about the whole situation, with what I think is a misleading headline. To me, the most interesting thing about this whole thing was the fact that there was an article at all. A telling statement:

The department could have decided to hold off submitting the data, but it might have appeared that police were trying to “cover up” the fact that violent crime surged dramatically in 2005, [the spokesperson] said. The department needed to meet a March deadline, she said.

There are several points I think we can take away from this article:

1. Information Quality is a high-visibility, political issue, even when handled correctly.
2. There is a quantifiable value in proactively and honestly addressing information quality issues.
3. Good organizations working together make dealing with information quality problems manageable.
4. Good people make all the difference in information quality.

I think need to clarify some things on my article about truthiness in information architecture. I may have come across as a bit of a luddite or somehow opposed to using information architecture. In reality, nothing could be further from the truth, so I think I may need to explain a bit of my philosophy.

While it may seem surprising to some, the nature of truth is still very much up for debate within the realm of philosophy. The two types of thought that seem to be at odds are foundationalism and coherentism.

A foundationalist believes that:

1. A complete philisophical system can be built from basic beliefs.
2. In a system, basic beliefs cannot be questioned.
3. Basic beliefs can be used to justify more beliefs, which can in turn be used to justify other beliefs, and so on and so on.
4. A belief is only justified if it is a basic belief or if it has been justified by a chain of beliefs.
5. A statement is true if it can be ascribed to a basic belief.

A coherentist believes that:

1. Truth is primarily a property of the entire philisophical system.
2. Any belief within the system can be questioned.
3. The truth of a statement can only be determined according to their coherence with the whole.

BTW, if this stuff interests you I would recommend taking a look at the Philosophy Portal at Wikipedia.

In my experience, I have found that most technologists subscribe to foundationalism, but most organizations function under coherentism. This difference in worldviews is a primary source of misunderstanding and conflict between the two parties, often on an unconscious level.

Based on this, I have come up with some ideas of my own …

Morgan’s Conjectures on Organizational Truth

1. Within for a given organization, it is entirely possible to have two or more different ideas of the “truth” for the same set of facts.
2. The quantity and variety of “truths” will increase as an organization grows.
3. Each of these “truths” may be valid and necessary for the organization to function on a basic level.
4. An information architecture that does not consider this will not be useable and efficient over the long-term.

This is getting a bit long, so I will discuss this more later …

technorati tags:information quality, data quality, etl, automation, design , truthiness

Steve Hoberman wrote an interesting article in the June 2006 DMReview. He runs the design challenge, an email-based community of data professionals that tries to come up with answers to difficult ETL and Database related problems. Design challenge #14 was somewhat interesting, and worth a look if you are so inclined. The challenge deals with the challenges around generating a unique identifier for an individual in a diverse data environment. For example, in an academic environment, the same person might be both an instructor and a student (think about the army of graduate students who will be TA’s for Psych 101 at any large institution) .

To be honest, I was not particularly impressed with the solutions that were suggested. Working with ETL and Data Modeling I think I see this situation on a regular basis. A lot of the suggestions seemed like overly complicated solutions that were more work than they were worth. However, there was a real gem in the discussion about the question itself:

As analysts and modelers, we continuously find ourselves asking the business, “Why?” For example, “Why do you need this report?” or “Why do you enter this information in three places?” Emma Fortnum, application architect, rightly asked why for this challenge as well. “What I would do is question if the university really needs a unique Person ID across all the roles. Do they really need to know that Bob the student is the same as Bob the instructor?” A holistic view of a person should only be created if there is significant business value. An organization-wide program such as a BI initiative or an enterprise resource planning implementation is usually the driver of the need for holistic view.

Hoberman (and Fortnum) are right on the mark here. My guess is that the requirement for a single ID per person came out of an ivory tower or a management meeting, and from someone who hasn’t had to deal with the hands-on design and implementation of a process in many years, if ever.
All too often, the highly touted, objective “Single Version of the Truth” is a concept that is simple, probably too simple for our subjective reality. Much as technologists abhor it (and Consultants are willing to bill hours to fight it), we live in a world where truthiness reigns, and we are the better for it. The “truth” for a given situation will be derived from subjective elements of data by subjective people with subjective interests from subjective data in a subjective model. Using an objective view of truth in the data warehouse is like using the movie Highlander as a model for your book club discussion. In the end it may work for whomever is left standing, but the process is long and bloody and ultimately isn’t worth the price.

technorati tags:information quality, data quality, etl, automation, automation

Just spotted this article on a $45M processing error made by ANZ Bank. Evidently, an ETL process was accidentally run twice. According to the bank’s spokesman:

I think what’s happened is the tape containing all of these transactions was run twice. We are just trying to get the bottom of why that’s happened but we suspect there’s some sort of human error involved.

This absolutely floors me, for several reasons. I find it hard to fathom that a mission critical process:

1. Was being run manually.
2. Did not prevent human error from effecting the overall system.

Now, I realize that bank transactions are inherently more complicated, as they move things closer to real-time processing. Also, I do give credit to ANZ for addressing the problems quickly, I am sure they are much chagrined and are looking at things very closely to see how this happened. Still, this is a big deal.

The next time you are thinking about the costs around a project to improve ETL, automation, or information quality, think about how much it is worth to your organization avoid embarrassment like this.

BTW, it might seem like I am picking on non-US organizations lately, please let me assure you that this isn’t the case. I it might just be that this gets written about more internationally.

technorati tags:information quality, data quality, etl, automation