Project X Discussions has an interesting article about project staffing and the bloat that often occurs with data related projects.

In a number of client environments I have often been amazed by the number of people that can be assigned to a project. Project Managers, Business Users, Business Analysts, Architects, Technical Analysts, Developers, Database Analysts, Database Administrators, Data Modelers, Subject Matter Experts, Testing specialists, Data Assurance people, Production and Operations People and of course a couple of people like me, the consultants.

While it is impossible for one person to do everything I often wonder how many people on such a project team could be removed from the project without impacting and perhaps improving the outcome.

They make some good points. In a structured environment it is very easy to get project bloat due to specialization and role playing. In my experience, a few smart, determined generalists deliver more than a pack of highly qualified specialists most of the time.

The Twist

The article then goes on:

One person I worked with recently proved that a single person acting alone can accomplish amazing things if they have access to all the right tools and know how to use them. In this scenario the end user need some reports - in excel format.

My friend, used UNIX scripts against flat files and SQL queries against the database to create a number of SAS data sets. He ran SAS functions against the data to aggregate it and exported it into Excel for the end user. It took him a couple of days.

It is easy to confuse project bloat with the need for high-quality business processes, and we have to be careful in this regards. I believe that this person was working with what I call “borrowed productivity”. He was able to deliver his part quickly, in large part by doing things that will make things more difficult (and expensive) in the future. He avoided all the processes that will save money over time with a quick fix immediately.

Where the organization will pay down the line will be:

1. Quality.
2. Sustainability.
3. Reusability.
4. Consistency.

My Experience

I once worked for a large (Fortune 500) company who had a division that did almost all of their reporting from Microsoft Access databases. They were able to crank out report after report and get them into excel format and delivered via email very quickly. At one point, this group was generating millions of emails

At the same time, there were a lot of things that didn’t work on this model:

• The reporting group was really never able to get their reports to match up internally, not to mention reports generated through the centralized data warehouse.
• When business processes or changed it was very difficult to find all the places where code needed to be updated.
• There were a huge number of reports that were very similar, with only slightly different parameters.
• When more reports were needed the only recourse was to hire more Analysts.
• The users soon grew tired of having a large number of excel spreadhseets in their Inbox each morning and stopped reading many reports

Eventually, this reporting system became unsustainable and the group went through several major crises and ended up being mostly disbanded. This was a huge waste of resources and a bitter loss of business expertise and technical talent. A lot of good people lost their jobs because of a bad system.

Conclusion
Small teams are good, very good. They are efficient and often much easier to run. Small thinking is bad, very bad. It often unintentionally deceptive and very expensive in the long run. There is no causation between big groups and small thinking or small groups and big thinking, but often there is a correlation.

Regardless of your environment, in developing your inormation architecture make sure that you understand that:

1. Data delivery has to be fast, at least as fast as the business that drives it.
2. Initial development is usually the smallest cost in the data lifecycle.

Normally, a discussion around this subject will neglect one side or the other, but this really is just a disservice to the efficiency of your organization.

Other Reading

Rick Sherman has discussed shadow systems , this is one of the few works I have seen that view the discussion holistically. I would highly recommend taking a look at his work.

I just wanted to let everyone know that I am going to be slowing down on the posts this week (and perhaps part of next), in order to do some planning and some re-tooling in order to make the site more effective.
In the upcoming months, I would like to have:

1. Better use of links and integration with del.icio.us.
2. A more rapid publishing schedule.
3. A greater focus on practical application and information quality.  This will probably include a continuing series on ETL, with a focus on practial application and Information Quality.

Thanks for your patience.  If there is anything that you can see that will improve the site, please leave a comment or drop me a line.

Morgan

The more important outcome of a decision, the more people will resist using evidence to make it.

– Michale Lovagalia, Chair of the Department of Sociology at the University of Iowa.

A little while back, Ingres and rPath announced Project Icebreaker the creation of a software appliance encompassing “the integration of the Ingres 2006 database with the Linux operating system”. In the past, I have always thought of appliances as being hardware related, but recent events have had me reconsidering this narrow definition. Under my old way of thinking I might have dismissed Icebreaker as something of a response to the difficulties of administering and maintaining a Linux system. However, some new developments have me thinking …

An applicance is the complete encapsulation of a business process, freeing the user to step back and look at the bigger picture. For a product like Netezza, this it pretty obvious that it is a complete RDBMS encapsulated in hardware black box. But, as hardware becomes more of a commodity, operating system configuration becomes more complicated, and virtualization becomes widespread, it makes sense that the definition of an appliance could migrate out of silicon.

Consider that Amazon Web Services are working on some very interesting projects, namely namely S3 (online storage) and EC2 (grid for rent). These projects have essentially abstracted the operation and maintenance of processing and storage. Icebreaker sounds ideal for this kind of enivronment. Thinking of the metaphor of the kitchen, it changes the role of the server from appliance to countertop.

OK, OK, this is just a jumble of random thoughts.  I have been trying all week to get these thoughts out and on to paper, pretty unsuccessfully.  The one thing I think I think is that there are some radical changes just over the horizon, based on the trends in web services, appliance-ization, and outsourcing.  Something big, something both familiar and new and unexpected.  I will have to think about this a bit more …