Working on Borrowed Productivity
Project X Discussions has an interesting article about project staffing and the bloat that often occurs with data related projects.
In a number of client environments I have often been amazed by the number of people that can be assigned to a project. Project Managers, Business Users, Business Analysts, Architects, Technical Analysts, Developers, Database Analysts, Database Administrators, Data Modelers, Subject Matter Experts, Testing specialists, Data Assurance people, Production and Operations People and of course a couple of people like me, the consultants.
While it is impossible for one person to do everything I often wonder how many people on such a project team could be removed from the project without impacting and perhaps improving the outcome.
They make some good points. In a structured environment it is very easy to get project bloat due to specialization and role playing. In my experience, a few smart, determined generalists deliver more than a pack of highly qualified specialists most of the time.
The Twist
The article then goes on:
One person I worked with recently proved that a single person acting alone can accomplish amazing things if they have access to all the right tools and know how to use them. In this scenario the end user need some reports - in excel format.
My friend, used UNIX scripts against flat files and SQL queries against the database to create a number of SAS data sets. He ran SAS functions against the data to aggregate it and exported it into Excel for the end user. It took him a couple of days.
It is easy to confuse project bloat with the need for high-quality business processes, and we have to be careful in this regards. I believe that this person was working with what I call “borrowed productivity”. He was able to deliver his part quickly, in large part by doing things that will make things more difficult (and expensive) in the future. He avoided all the processes that will save money over time with a quick fix immediately.
Where the organization will pay down the line will be:
- Quality.
- Sustainability.
- Reusability.
- Consistency.
My Experience
I once worked for a large (Fortune 500) company who had a division that did almost all of their reporting from Microsoft Access databases. They were able to crank out report after report and get them into excel format and delivered via email very quickly. At one point, this group was generating millions of emails
At the same time, there were a lot of things that didn’t work on this model:
- The reporting group was really never able to get their reports to match up internally, not to mention reports generated through the centralized data warehouse.
- When business processes or changed it was very difficult to find all the places where code needed to be updated.
- There were a huge number of reports that were very similar, with only slightly different parameters.
- When more reports were needed the only recourse was to hire more Analysts.
- The users soon grew tired of having a large number of excel spreadhseets in their Inbox each morning and stopped reading many reports
Eventually, this reporting system became unsustainable and the group went through several major crises and ended up being mostly disbanded. This was a huge waste of resources and a bitter loss of business expertise and technical talent. A lot of good people lost their jobs because of a bad system.
Conclusion
Small teams are good, very good. They are efficient and often much easier to run. Small thinking is bad, very bad. It often unintentionally deceptive and very expensive in the long run. There is no causation between big groups and small thinking or small groups and big thinking, but often there is a correlation.
Regardless of your environment, in developing your inormation architecture make sure that you understand that:
- Data delivery has to be fast, at least as fast as the business that drives it.
- Initial development is usually the smallest cost in the data lifecycle.
Normally, a discussion around this subject will neglect one side or the other, but this really is just a disservice to the efficiency of your organization.
Other Reading
Rick Sherman has discussed shadow systems , this is one of the few works I have seen that view the discussion holistically. I would highly recommend taking a look at his work.
