A couple of articles about bad data entry and incorrect data transfers got me thinking about sanity checks. Bad data entry is a fact of life, but when someone is deployed to Iraq based on a clerical error (especially when they already knew there were issues), that is a cause for concern. Bad file transfers happen as well, but when crime victims get phone calls falsely notifying them of the release of the offender, it becomes a concern.

A sanity check is a simple, usually external way to ensure that the data operation you are about to do is actually worth doing. For example, you might have a 3rd party vendor send you a data file every night that contains information about your customers. Some simple sanity checks might be:

• The file should not be empty.
• The file should not be more than 10 MB.
• The file should not be sent more than once in a 24 hour period.
• The file should arrive between XXXXX and YYYY.

I think that sanity checks are the most simple and cost effective ways to improve your information architecture, especially with batch processes, reporting, transformations, cross-system interactions, and data transfers. Now, this just scratches the surface information quality. However, the right sanity checks are will tell you when something is probably wrong. Think of it as that hunch that tells you that something isn’t quite right (”It’s quiet … too quiet“). The consultant in me wants to write a whitepaper and a presentation touting the value of “high-reward requirements instantiations” or some such jibberish, but I will refrain.

I like sanity checks for several reasons:

1. They are low-risk/high-reward, truly the low-hanging fruit of quality improvement.
2. They are usually very easy to understand and cheap to implement.
3. It is easy to communicate their value.
4. They will prevent you and your organization from getting egg on your face.
5. It is usually fairly painless to standardize them and use them for all processes.
6. They can open up a much larger discussion on information quality.
7. After the first time something blows up publicly, someone is going to ask why this wasn’t already in place anyway.

Consider if there are any processes that you are using that need (or need better) sanity checks. I guarantee you that it will be worth the time and effort.

technorati tags:automation, ETL, information architecture, information quality

This entry was posted on Sunday, July 9th, 2006 at 5:37 pm and is filed under ETL, Information Architecture, Systems Integration, Information Quality, Practices. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.