In the IT world, I am not sure if I should take user-driven innovation to be a sign of progress or a sad display on how difficult things have become. With all the talk about Web 2.0, wikis, social media, and the cathedral and the bazaar, I would think that any technologist who isn’t at least passingly familiar with these ideas must be on a contract, working out of a cave, typing programs on dusty green screen terminal connected to the corporate mainframe by a thick token ring cable.

However, the ‘new trend’ of allowing users actually provide feedback into the products that they are using has produced a professorship at MIT and been mentioned in the New York Times ::sigh::  IMHO, this is simply the realization that ignoring people is less effective than communicating with them, and less effective means less profitable.  I think the technology world should be embarassed that it is so disfunctional that this is at all new or interesting enough to study.

On a similar note, Ben Stein has some interesting thoughts about how to have a business conversation.  If you plan on doing anything other than pure solo work for the rest of your life it is mandatory reading.  Interestingly, it seems that the best conversationalists are the ones who are the most considerate and do the most listening.  Sounds like organizations might want to focus on having an actual conversation with their customers …

GigaOM has an interesting article about the impact of web 2.0 on network engineers. Namely, that the maturation of the internet has made the skills of a good network person a lot less important:

I see the current state of the Internet as the ultimate success … You can deploy a wildly successful Web 2.0 application that serves millions of users and never know how a router, switch or load-balancer works. Even network security and firewalls that were making headline news not more than a few years ago are considered perfunctory. The success of these networking devices and technologies has enabled them to become part of the technology landscape that exists for all to use as they see fit, similar to the microprocessor or electricity.

It is always odd to see the once-glamorous jobs of your youth thrown onto the scrap heap of history (think about the differences in perception between the masons of the middle ages and your local bricklayer). Network Engineers were once the masters of a difficult and arcane field, literally bringing information from chaos. Now, the wizards have been trapped in tiny control panels for now, until they can be embedded in silicon for all time.

This has really got me to thinking about my own field, and its future. What specialties are going to dissapear if data becomes as reliable as electricity? For one thing, I think we would see ETL and Business Analysis become a single career path that is much more abstract and tools-based. With the advent of good BPM, I could see a lot of the scheduling and other mechanics pushed off towards the DBA’s and Systems Administrators. Also, I think that a lot of the hardware could be appliance based, or outsourced completely. Of course, this leaves a great opportunity for open source BI and for nimble players to attack the market and take advantage of the innovators dilemma.

A brave (an infinitely more useful) new world!

A while back, I wrote about how useful it would be to be able to combine collaborative information and structured data. While Wikis and other collaborative information sources are great, I would argue that they aren’t useful until they can be used in aggregated or statistical form for strategic decision making or automation. Until then, they are too “abstract” to be useful (at least in the mechanical sense of the word).

Recently, ran across DBPedia, an organization that is turning Wikipedia entries into RDF, the language used for the semantic web. DBPedia has actually has downloadable datasets based on Wikipedia that are available today. These are datasets that can be queried with existing tools and linked to other datasets. Wow!

Even if you aren’t a Wikipedia fan, this is really a big step forward for the enterprise. Think about the amount of knowledge that exists in your organization that isn’t captured, but is critical to your operations. It has always been a big pain to try and sit down and do formal knowledge engineering. However, most people are comfortable enough with a Wiki to sit down and start typing. For a small organization this might not be such a big deal, but for a larger enterprise this could provide some very useful information.

The first time your Director or CXO can make a financial decision based exclusively on the information from your company wiki, it will have proved its worth. Until then, it is just another trendy tool. The work that DBPedia is doing is an important step in making this a reality.

OK, when USA Today has a story about information management you can be sure that the phenomenon is big enough that it will impact non-techies on a large scale.

When tech analyst John Gantz at researcher IDC began tallying up all the digital information generated annually, he first looked in the obvious places …

Gantz ultimately calculated that 161 exabytes of digital data — or about 161 billion GB — were generated in 2006. And the amount is expected to rise fast.

It is worth a quick read, although if you are a data geek a lot of this is pretty elementary, so you might just want to forward the URL on to your favorite business leader or project manager.

The article is a bit light on is the downstream ramifications of the data deluge. Obviously, there will be a lot of SAN units sold, but that opportunity is long gone unless you are able to take advantage of the innovators dillema. Looking over the horizon, there is a huge opportunty for people who can make this data not just searchable, but accessible and usable and trustworthy for decision-making.

The Sarbanes-Oxley Compliance Journal has a very detailed article discussing the implications of spreadsheets on compliance and financial regulation. It is a well thought out, well written piece that gives very specific, very conservative steps to ensure that an organization doesn’t run into regulatory issues from day-to-day business practices.

One thing I appreciated in the article was its realistic tone, which recognized the existence of shadow systems and the role that they play in the real world. While the article focused on spreadsheets, this doesn’t mean that they are bad and that centralized CRM is good (as a matter of fact I might argue the opposite in many cases). The real issue being discussed is the risk that occurs when information architecture doesn’t match the needs of the organization.

Looking Globally

For many low-importance, one-off operations, a spreadsheets is fine. It can be shared easily, and usually owned by one person. However, what normally happens is that someone sharp (and usually not someone in IT) decides to do try to build some infrastructure around a spreadsheet without considering the consequences. It is quick, cheap, and easy to do, to a point. The problem is that the point when a spreadsheet becomes unmanageable is often well after the point where an organization depends on its output to function effectively.

For any system, an information architect needs to consider:

• Cost
• Effectiveness
• Visibility
• Traceability
• Mangageability
• Quality
• Auditability

Furthermore, the situation needs to be considered from a forward-thinking perspective. That is, we need to try to understand how the landscape is going to look a few years down the road, and to make our systems flexible enough so that we don’t put our organization into a bind if we are wrong.

Spreadsheets are normally built only considering cost and effectiveness, something that is only discovered after time has gone on. It is often the case that a system would be much more effective living in a database, BI tool, or custom software application when considered over time.