Architected Information

Musings on Metadata and Compliance

morgan — Fri, 29 Sep 2006 12:10:22 +0000

Frank Dravis has a new post on metadata and its growing importance in the marketplace. Dravis wonders how metadata became a subject of interest for non-technical folk …

In years past, metadata was the domain of data architects. It helped them understand what data they had and how it related to the sources and operations from which it came and to which it went. At the first mention of metadata business users would roll their eyes and head for the conference room door. Surely metadata was the stuff of arcane IT discussions best had out of earshot of people driving and running the business.

Then metadata management progressed and someone had the silly idea of articulating the business value, the value to the business side of the house, for metadata. The value came from the resolution of an age old problem. A corporate manager is sitting in a conference room looking at their regular monthly sales report and it is different from what they expected based on anecdotal evidence from the field: the numbers are too low.

Personally, I think that this recent interest is driven by a few things:

Regulation and the threat of real penalties for inaccuracies in reporting. People got interested enough to protect their own hides.
The rise of ERP and BPM in the marketplace. If everything is in one place then metadata suddenly becomes a lot easier to manage.

Truthfully, I wonder how all of this is going to turn out. I know there are lots of people who want to sell metadata software, but in my experience it takes a lot of resource (time, effort, and expertise) to maintain a comprehensive metadata environment. The threat of jail time helps to keep people motivated enough to save their necks, but not enough to make something useful. Being locked into an ERP package can mean the same thing, only it is your data that is locked.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

Case Study — Statistical Information Quality

morgan — Mon, 14 Aug 2006 13:51:32 +0000

Introduction

When an organization begins a concerted effort to improve its information quality, often it gets stuck in trying to figure out exactly where to start. This case study takes this to heart and gives a specific example of an approach to improving information quality.

Previously, we had discussed the semantic and statistical approaches to information quality and linked them to black box and white box testing. In addition, there is a case study on semantic information quality which is used to contrast this case study (you may want to take a look at these if you aren’t familiar with the subjects).

The example that we have been using is …

dealing with call data for a customer contact center. For simplicity, we can assume that all the call data we need is delivered nightly and is loaded into a single table that looks exactly like the files as they have arrived. This table has the following attributes:

employee_login_number

site_name

department_name

call_local_start_time

call_local_end_time

… from this data the business analysts are going to figure out how much to pay and to whom. Also, we need to figure out who is handling the highest call volume (vendors, locations, and employees) on a daily basis so that we can resolve issues and negotiate contracts. Our job is to make sure that the data is accurate enough to do this with confidence.

Also, before we get started, realize that with the semantic and statistical approaches we are trying to do the same thing in different ways. So, while we are doing things differently, there is bound to be some overlap.

The Statistical Approach

With a statistical approach, there are several things to consider:

From a statistical point of view, there is nothing special about this dataset. It has very similar characteristics to all the ones that came before it and will come after it. We should try to create an architecture that can be re-used where appropriate.
There is a lot that we can infer from the dataset itself. We can learn a great deal of information about the dataset very cheaply through black box testing. Focusing on these areas will maximize re-use as well.
We can probably assume that any data that we recieve is of reasonably good quality when the process was first designed. Therefore, we can focus on events where the nature of the data changes substantially.

With these in mind, we can start to design a solution.

The place to start is to ask, “what can go wrong in our data?”. I can think of several situations that might impact the quality of this data:

The employee_login_number is invalid or NULL.
The site_name is invalid or NULL.
The department is invalid or NULL.
The call_local_start_time is invalid or NULL.
The call_local_end_time is invalid, NULL, or starts before the call_local_start_time.
Due to errors outside of our control, the process that created the data malfunctioned. Often, this will show up as duplicate values, irregular frequency or distribution of values

Off the top of my head, I have a number of questions about the data that we will see day to day:

For each column, is there a distinct list of values (call this the domain) that are valid?
For each column, is there a distinct pattern of values that are valid?
For each column, can the values be NULL?
Is there a distinct key? If so, is it unique?
For column values and keys, should the frequency for particular values be fairly normal?
Is there a certain number of rows that should be expected (by key or for the entire dataset)?
Is there a certain number of keys that should be expected?
For numeric values, can we do descriptive statistics to tell us if things are off-kilter?

Based on these, I think that we can establish a data model that would allow this metadata to be recorded for multiple processes, which would allow it to be used for reporting and decision-making.

For example, consider a table having the following attributes:

process_id
process_run_dt
distinct_value
distinct_value_type
distinct_value_count

This would allow the user to keep track of how many distinct values there were generated by a given process. Over time, this could be very useful in tracking down some sticky problems, and perhaps prevent bad data from ever getting into a data store in the first place.

For each of the meausurement processes we mentioned, they can probably be integrated into the overall data model in a process agnostic way. I apologize for not having more details at this point, I plan to move this to the wiki (at some point) and put in a reference model for doing some of these operations.

Comparisons With Data Profiling

For people with some experience with data management this may sound a lot like data profiling. In fact, a lot of the operations inherent in the statistical approach would probably be considered a part of data profiling as well.

However, there are some key differences between Statistical IQ and Data Profiling that need mentioning:

Statistical IQ has an operational focus and needs to be as lightweight as possible. We want to use this to make day-to-day operational decisions about our data without slowing anything down.
Statistical IQ does not include data discovery, while data profiling often does.
One of the core functions of data profiling is establishing relationships between datasets. Statistical IQ has a very limited view of relationships in order to maximize functionality and reusability.

Similar base concepts, focusing on different areas.

Statistical IQ and Mad Libs

One thing that often gets lost in the re-use discussion is the price of user configuration. All too often, programmers push too much decision making out of their code and on to the operator, making it difficult to use.

The trick with Statistical IQ is that you have to be able to tie a generic statement (”there are 15 distinct values in this dataset”) back to something useful (”there is probably missing data, don’t continue the process”). While this might seem like a challenge, it can be done without a lot of heartburn.

In a recent engagement, I designed a solution where we tied every possible error back to an english description of the problem that was stored in an SQL database. This was done in a very generic way, so that new errors could be added or removed without any configuration required by the developer or operator.

Conclusion

There are different approaches to information quality, each with their own strengths, weaknesses, and costs. The statistical approach is cheaper (especially when you factor in Moore’s Law), but gives a less detailed picture of overall quality. The semantic approach is more expensive, but can be as comprehensive as the situation requires. A balanced approach will use both approaches to deliver the solution that is needed.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

Case Study — Semantic Information Quality

morgan — Fri, 04 Aug 2006 20:27:08 +0000

Previously, we had discussed the semantic and statistical approaches to information quality and linked them to black box and white box testing (you may want to take a look at these if you aren’t familiar with the subjects, as these are the basis for this article).

The Semantic Approach

A more semantic approach would involve defining exactly what your data represents, and from there determine what it should look like and how it should behave. This sounds pretty easy, right? The problem is that things are often more complicated than they seem.

Let’s look at an example I ran into on a client engagement, dealing with call data for a customer contact center. For simplicity, we can assume that all the call data we need is delivered nightly and is loaded into a single table that looks exactly like the files as they have arrived. This table has the following attributes:

employee_login_number
site_name
department_name
call_local_start_time
call_local_end_time

OK, now from this data the business analysts are going to figure out how much to pay and to whom. Also, we need to figure out who is handling the highest call volume (vendors, locations, and employees) on a daily basis so that we can resolve issues and negotiate contracts. Our job is to make sure that the data is accurate enough to do this with confidence.

The Semantic Challenge

The first thing we would need to do is to find out exactly what is going on in the system. Talking with various people in technology and business units, we can define some basic terms. In our case, let’s say that we discover:

There are multiple contact center locations worldwide and each one has its own “switch” with data in its own local time. All of the locations are owned and operated by vendors.
All reporting for management is done in Eastern Time (US), but location and employee reporting should be done in local time.
An agent is signified by a login number in the “switch” (a piece of telephony equipment).
An agent works in a department, which handles a specific type of call.
An agent can have multiple logins on the same “switch” for different departments that they work in.
A “call” will be defined by a valid call record, including a start time and end time
Each time a call comes in a record will be created with the login number, start time, and stop time (in local time).
Calls to different departments are paid different rates.

Realize that this is the tip of the iceberg when it comes to business rules. There could easily be 100 more concepts and constraints involved in a decent sized business. Also, understand that this was very rapidly growing (over 100% per year) worldwide business that was intensely focused on customer service. We couldn’t ask the business to slow down. But, we still needed to provide data that was of high quality.

From the problem description, we know that there must be mappings between:

Logins and agents.
Locations and vendors.
Locations and time zones.
Departments and pay rates.

The Semantic Solution

Off the top of my head, there are a number of things that we can do to test this data. It shouldn’t be too hard to write SQL that would test the referential integrity of the system. For example:

Join the call data with each of the mappings, noting what records have no matches.
Join the call data with each of the mappings, noting what records have multiple matches.
Look for duplicates in the mapping tables.
Look for newly added or removed values in the mapping tables.

Next, I would look at some basic validation tests:

Each agent should not have more than 3 logins (or some appropriate number) per day.
Each agent should only be listed at one facility per day.
Each agent should only be listed at one vendor per day.
Locations should not disapear or change time zones from day to day.
Vendors should not disapear from day to day.
The Call Start Time should be earlier than the Call Stop Time.

Last, it would be good to write some sanity checks:

After daily processing is complete, the total number of calls should exactly match the sum of the number of calls to each site.
After daily processing is complete, the total number of calls to a site should exactly match the sum of the number of calls to each agent at that site.
At all levels, the total amount billed should not be more than the total (number of calls) x (highest billing rate).
The total number of call time for an agent should not be more than 12 hours in a given day.

Now, this is by no means a complete list of tests that should be run, but it gives you a good idea of what can be looked at.

Conclusion

As you can see, ensuring that the information coming out of this process is accurate is sometimes simple, sometimes complex, and sometimes downright daunting. Most of the solution here requires that custom tests be created, maintained, understood, and reported on (something that we haven’t even discussed). This is a lot of work, and customized work that can’t be easily reused. This is why in most cases I believe this type of testing is only added after an issue had occured.

In part 2, we will discuss a statistical approach to the same dataset.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

Two Methods for Defining Information Quality

morgan — Tue, 01 Aug 2006 03:03:57 +0000

In Information Science today two competing methods for indexing information: semantics and statistics. While this may not seem to have a lot to do with information quality, bear with me and I promise I will link them up (eventually). Both methods approximately the same job, that is to allow information to be read and manipulated by machines on a grand scale. The difference is in how this is done.

A semantic approach would have the author define concepts and relationships ahead of time. You can see some examples in this tutorial, as they are long and would be difficult to reproduce here. The Semantic Web would be a good example of this methodology.

A statistcal approach would simply look at the text that was available and try to determine what is there and how it relates to other things through textual analysis and aggregation. Google is a good example of the use of this approach.

The semantic way of looking at things is very abstract and much more rigorous. It says that there is a truth to be represented, it designs a way of doing it, and expects everyone to follow along. The statistical way of looking at things is much more flexible. It says that there are things to be gleaned regardless of form, and that we should accept this fact and try to make the best of things. Not surprisingly, the semantic approach is the favorite of academia and has been under development for many years, while the statistical approach is already in real-world use.

What got me thinking about this in the first place was the latest issue of Baseline. Specifically, it was an article from Paul A. Strassman titled, “How Clean Data Can Transform Your Business”. Normally Strassman’s stuff is pretty good, but it is helpful to note that Strassman is a senior consultant to the Department of Defense and has been in the business for a long, long, long time.

The crux of his argument was that:

The first step in business transformation: enterprisewide standardization of data. That calls for the declaration of a metadata directory as the template for defining data that can circulate within a firm’s information systems. The policy and implementation of an enforceable metadata directory likely will be resisted by bureaucrats, who see this as a threat to their indispensability. It will not be welcomed by systems developers, contractors and vendors, who prefer to concentrate on upgrading software as a technologically more interesting—and profitable—task.

A classic argument for a semantic model of truth. We just need to get everything defined and then it will be smooth sailing from there. For most vendors and consultants, the semantic view is the accepted one, probably because it is so structured and logical, although at least partially because it all those hours spent defining concepts are billable. Even Strassman acknowledges this reality …

To reach agreement on the representation, semantics and taxonomy of data, you will likely go through a painful political process that must be adjudicated by line management. This can get messy because it will reveal that a large percentage of installed software perpetuates incompatible, unreliable, insufficiently secure and delayed information.

With this in mind, is semantic definition the most efficient way to improve information quality? Is a statistical definition the most descriptive way to understand information quality? We will explore the basis for both of these methods in the next part of this series.

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.

The Information Quality Pyramid

morgan — Tue, 25 Jul 2006 03:18:11 +0000

I have been working on some more detailed articles for the wiki to help illustrate some ideas about information quality. While I don’t want to just duplicate that article here, I thought I would post some things on the blog and get some feedback.

I am currently working on the Information Quality Pyramid, which discusses the various components that go into improving information quality across an organization:

The pyramid is made up of several parts, each of which are important in their own right. However, the base components (in blue in green) have the interesting combination of being very important, terribly inexpensive, and totally unglamorous.

Understanding Your Organization – The single most important thing that you can do to ensure success in any information quality effort. Without a solid understanding of how your organization works it is virtually guaranteed that you will not be able to deliver the solution your customers need. This (coupled with the need for extreme customization) is one of the reasons that it is very difficult to outsource this type of work.

Architecture and Design Practices – To put it bluntly, if you build your information architecture in an inconsistent manner then you have to expect inconsistencies in its output. These inconsistencies become quality-related issues very quickly. If you can proactively address (or at least mitigate issues around) consistency through your architecture then you can dramatically improve the quality of information that you produce.

Automation – The key to high-value, high-quality information architecture is automating everything possible. This is because:

Moore’s Law will double the speed of computerized processes every two years. It is pretty tough for humans to keep up.
Humans make mistakes.
In order to automate a process, it has to be understood by more than just the designer or the developer.

Sanity Checks – The easiest and most cost effective ways to catch issues before they become problems.

Data Profiling – The only way to understand your information is to know your data. Intimately. Regularly. Historically. Profiling takes a generic look at an arbitrary dataset and discovers important statistical information about it. Profiling is by far the cheapest and most reliable way of examining data (think of it as an expanded sanity check).

Process Testing – Instead of looking at a dataset in a generic way, process testing looks at things in a very specific way. These should be customized tests that will tell information that are automated and deliver results that are unique to the process. Because of the level of customization and effort, this is significantly more expensive than profiling or sanity checks.

Human Intervention – Anything that involves humans, from adjusting processes already in production to performing manual analysis to resolve concerns to creating new code. Think of it as if all of information quality was outsourced to a 3rd party company and all personnel costs came directly out of your budget. This is the true cost of IQ, it is just that people see it in a more abstract sense.

The one category that I can see people might think is missing here is metadata. I think metadata is an incredibly important part of information quality, but I tend to value it in its most concrete form instead of in the abstract. I will get into this more in the wiki article.
Any feedback would be most appreciated!

technorati tags:information architecture, information quality, data quality, automation, metadata

Share and earn some karma ...These icons link to social bookmarking sites where readers can share and discover new web pages.