I see the current state of the Internet as the ultimate success … You can deploy a wildly successful Web 2.0 application that serves millions of users and never know how a router, switch or load-balancer works. Even network security and firewalls that were making headline news not more than a few years ago are considered perfunctory. The success of these networking devices and technologies has enabled them to become part of the technology landscape that exists for all to use as they see fit, similar to the microprocessor or electricity.

It is always odd to see the once-glamorous jobs of your youth thrown onto the scrap heap of history (think about the differences in perception between the masons of the middle ages and your local bricklayer). Network Engineers were once the masters of a difficult and arcane field, literally bringing information from chaos. Now, the wizards have been trapped in tiny control panels for now, until they can be embedded in silicon for all time.

This has really got me to thinking about my own field, and its future. What specialties are going to dissapear if data becomes as reliable as electricity? For one thing, I think we would see ETL and Business Analysis become a single career path that is much more abstract and tools-based. With the advent of good BPM, I could see a lot of the scheduling and other mechanics pushed off towards the DBA’s and Systems Administrators. Also, I think that a lot of the hardware could be appliance based, or outsourced completely. Of course, this leaves a great opportunity for open source BI and for nimble players to attack the market and take advantage of the innovators dilemma.

A brave (an infinitely more useful) new world!

Recently, ran across DBPedia, an organization that is turning Wikipedia entries into RDF, the language used for the semantic web. DBPedia has actually has downloadable datasets based on Wikipedia that are available today. These are datasets that can be queried with existing tools and linked to other datasets. Wow!

Even if you aren’t a Wikipedia fan, this is really a big step forward for the enterprise. Think about the amount of knowledge that exists in your organization that isn’t captured, but is critical to your operations. It has always been a big pain to try and sit down and do formal knowledge engineering. However, most people are comfortable enough with a Wiki to sit down and start typing. For a small organization this might not be such a big deal, but for a larger enterprise this could provide some very useful information.

The first time your Director or CXO can make a financial decision based exclusively on the information from your company wiki, it will have proved its worth. Until then, it is just another trendy tool. The work that DBPedia is doing is an important step in making this a reality.

When tech analyst John Gantz at researcher IDC began tallying up all the digital information generated annually, he first looked in the obvious places …

Gantz ultimately calculated that 161 exabytes of digital data — or about 161 billion GB — were generated in 2006. And the amount is expected to rise fast.

It is worth a quick read, although if you are a data geek a lot of this is pretty elementary, so you might just want to forward the URL on to your favorite business leader or project manager.

The article is a bit light on is the downstream ramifications of the data deluge. Obviously, there will be a lot of SAN units sold, but that opportunity is long gone unless you are able to take advantage of the innovators dillema. Looking over the horizon, there is a huge opportunty for people who can make this data not just searchable, but accessible and usable and trustworthy for decision-making.

The site is a comprensive look at a huge number of different ways to visualize information, ranging from the artsy (such as the graphic facilitation) the the mundane (pie chart) to the technical (entity-relationship diagram) to the silly (knowledge map).

The UI is very good, and examples of each different type of visual tool, and a nice overall classification of the different types of visual tools. This could be very helpful if you are stuck trying to figure out how to communicate information but don’t know exactly where to start.

Take a gander …

It is an interesting idea, where you …

1. Upload your dataset to Swivel.
2. Share your data with other users.
3. Chart your data, making it available on the web.
4. Compare your data with the datasets from other users.

Interested? Here is a reasonable example and a not-so-reasonable example of what Swivel can do.

It was a little bid odd to me, at first. Why would I want to provide my data to a web site so that I could look at it the same way that I can on my own computer? Well, I can think of a couple of reasons …

Some Analysis

First, you can share your data with anyone who wants it. This sharing takes two parts, display and anaysis. You can display your data on the web, for everyone to see. Second, you can combine your data with the data that other people have uploaded and perhaps learn something that you didn’t already know.

Now, sharing might be good, and it might not be, depending on your point of view. An enterprise might want to keep it’s information secret, and that makes sense. However, an interesting thing about most data is that it is subject to the network effect, big time. Two unconnected data points might mean something, but you can’t really be sure. However, a hundred data points indicate a trend or a correlation.

Second, you can use a community to learn more about your own data, as well as your correlations. At the bottom of each page, there is a rating that gives a group evaluation of how related and comments about how useful the comparison actually is.

Overall

A site like this could be useful, especially if there could be a concerted effort to provide good information, such as the census, demographic, and geographic information from the US government. It is still in its very rough stages, and it hard to say how much value will come out of combining disparate data sets. Also, I security, information quality, and analysis will be a huge issue here. Still, an interesting idea.