I see the current state of the Internet as the ultimate success … You can deploy a wildly successful Web 2.0 application that serves millions of users and never know how a router, switch or load-balancer works. Even network security and firewalls that were making headline news not more than a few years ago are considered perfunctory. The success of these networking devices and technologies has enabled them to become part of the technology landscape that exists for all to use as they see fit, similar to the microprocessor or electricity.

It is always odd to see the once-glamorous jobs of your youth thrown onto the scrap heap of history (think about the differences in perception between the masons of the middle ages and your local bricklayer). Network Engineers were once the masters of a difficult and arcane field, literally bringing information from chaos. Now, the wizards have been trapped in tiny control panels for now, until they can be embedded in silicon for all time.

This has really got me to thinking about my own field, and its future. What specialties are going to dissapear if data becomes as reliable as electricity? For one thing, I think we would see ETL and Business Analysis become a single career path that is much more abstract and tools-based. With the advent of good BPM, I could see a lot of the scheduling and other mechanics pushed off towards the DBA’s and Systems Administrators. Also, I think that a lot of the hardware could be appliance based, or outsourced completely. Of course, this leaves a great opportunity for open source BI and for nimble players to attack the market and take advantage of the innovators dilemma.

A brave (an infinitely more useful) new world!

Recently, ran across DBPedia, an organization that is turning Wikipedia entries into RDF, the language used for the semantic web. DBPedia has actually has downloadable datasets based on Wikipedia that are available today. These are datasets that can be queried with existing tools and linked to other datasets. Wow!

Even if you aren’t a Wikipedia fan, this is really a big step forward for the enterprise. Think about the amount of knowledge that exists in your organization that isn’t captured, but is critical to your operations. It has always been a big pain to try and sit down and do formal knowledge engineering. However, most people are comfortable enough with a Wiki to sit down and start typing. For a small organization this might not be such a big deal, but for a larger enterprise this could provide some very useful information.

The first time your Director or CXO can make a financial decision based exclusively on the information from your company wiki, it will have proved its worth. Until then, it is just another trendy tool. The work that DBPedia is doing is an important step in making this a reality.

When tech analyst John Gantz at researcher IDC began tallying up all the digital information generated annually, he first looked in the obvious places …

Gantz ultimately calculated that 161 exabytes of digital data — or about 161 billion GB — were generated in 2006. And the amount is expected to rise fast.

It is worth a quick read, although if you are a data geek a lot of this is pretty elementary, so you might just want to forward the URL on to your favorite business leader or project manager.

The article is a bit light on is the downstream ramifications of the data deluge. Obviously, there will be a lot of SAN units sold, but that opportunity is long gone unless you are able to take advantage of the innovators dillema. Looking over the horizon, there is a huge opportunty for people who can make this data not just searchable, but accessible and usable and trustworthy for decision-making.

Most interesting to me was their grid service, which provides an on-demand capability for web hosting that allows a site to handle the slashdot effect without having to blink an eye. Sites (and their corresponding media and applications) are running on multiple servers, which allows traffic to be spread out seamlessly, allowing for spikes in service and usage. This is all done without significant additional configuration, which makes it all the more sweet.

Now, they have had some problems, especially with non-grid oriented applications. However, I think that these are pretty minor compared to the utility that high-performance sites will get from using a grid environment.

Absolutely worth a look …

One thing I appreciated in the article was its realistic tone, which recognized the existence of shadow systems and the role that they play in the real world. While the article focused on spreadsheets, this doesn’t mean that they are bad and that centralized CRM is good (as a matter of fact I might argue the opposite in many cases). The real issue being discussed is the risk that occurs when information architecture doesn’t match the needs of the organization.

Looking Globally

For many low-importance, one-off operations, a spreadsheets is fine. It can be shared easily, and usually owned by one person. However, what normally happens is that someone sharp (and usually not someone in IT) decides to do try to build some infrastructure around a spreadsheet without considering the consequences. It is quick, cheap, and easy to do, to a point. The problem is that the point when a spreadsheet becomes unmanageable is often well after the point where an organization depends on its output to function effectively.

For any system, an information architect needs to consider:

• Cost
• Effectiveness
• Visibility
• Traceability
• Mangageability
• Quality
• Auditability

Furthermore, the situation needs to be considered from a forward-thinking perspective. That is, we need to try to understand how the landscape is going to look a few years down the road, and to make our systems flexible enough so that we don’t put our organization into a bind if we are wrong.

Spreadsheets are normally built only considering cost and effectiveness, something that is only discovered after time has gone on. It is often the case that a system would be much more effective living in a database, BI tool, or custom software application when considered over time.