OK, when USA Today has a story about information management you can be sure that the phenomenon is big enough that it will impact non-techies on a large scale.

When tech analyst John Gantz at researcher IDC began tallying up all the digital information generated annually, he first looked in the obvious places …

Gantz ultimately calculated that 161 exabytes of digital data — or about 161 billion GB — were generated in 2006. And the amount is expected to rise fast.

It is worth a quick read, although if you are a data geek a lot of this is pretty elementary, so you might just want to forward the URL on to your favorite business leader or project manager.

The article is a bit light on is the downstream ramifications of the data deluge. Obviously, there will be a lot of SAN units sold, but that opportunity is long gone unless you are able to take advantage of the innovators dillema. Looking over the horizon, there is a huge opportunty for people who can make this data not just searchable, but accessible and usable and trustworthy for decision-making.

Media Temple is a very cool evolution in remote hosting. Part web host, part application server, part grid, it is an interesting look into the parallell computing world we are rapidly moving into.

Most interesting to me was their grid service, which provides an on-demand capability for web hosting that allows a site to handle the slashdot effect without having to blink an eye. Sites (and their corresponding media and applications) are running on multiple servers, which allows traffic to be spread out seamlessly, allowing for spikes in service and usage. This is all done without significant additional configuration, which makes it all the more sweet.

Now, they have had some problems, especially with non-grid oriented applications. However, I think that these are pretty minor compared to the utility that high-performance sites will get from using a grid environment.

Absolutely worth a look …

Openfount has released S3Infindisk for EC2, a product that answers a lot of wishes in the EC2 community. One of the biggest issues with EC2 is the lack of persistent storage on the server instances, and this tool is a first attempt at a solution.

Basically, when you are using EC2, everything on your server that isn’t statically defined before the machine runs goes bye-bye as soon as the machine reboots. Not a real problem for software, but for data this is a major bummer. Amazon makes S3 storage available without data transfer fees, which is wonderful. However, it takes real effort to transfer data back and forth between the systems (with something like jsh3ll), and most data-centric tools (especially database servers) expect real-time access to a working file system.

S3Infinidisk bridges this gap, allowing an EC2 instance to use S3 like a real Linux filesystem. While it is a bit of hack, it allows data tools to work the way they need to and it allows an EC2 instance to take full advantage of the AWS environment. This is a huge step forward in making EC2 a more usable environment for utility computing!

I haven’t had a chance to try the product out yet, but am excited to do so. I appreciate the licensing structure (free single-user version + commercial high-performance version), although I would prefer seeing open source. Also, since the tool is based on the FUSE subsystem, I could easily see this spreading like wildfire.

I have been working with Google Docs and Sheets lately, in order to avoid the portability problem when working at different machines and locations. While it isn’t as fully featured as Excel, it does just about everything I need it to do, and then some. Plus, it adds in the collaboration features that are almost more useful to an internet-oriented business.

It would be incredibly boring for Google simply to replicate Excel and Word in a web format, unless you are an HTML groupie. However, there are some very, very interesting features that I think really turn the traditional office application on its ear. The first thing that caught my eye was the Google Lookup function, which allows one to incorporate search information dynamically into documents. The second thing was the Google Finance function, which allows financial information to be leveraged as well.  The third thing was the ability to embed portions or entire spreadsheets into a blog or web page.
Very cool stuff, and very interesting results. One could imagine this type of thing being leveraged with Froogle, Maps, or other service, within a document, presentation, or spreadsheet. Low cost, high reward stuff. However, there are some ramifications with using this type of information. For example, the spiffy new spreadsheet you put together for your boss could be modified by outside influences (like a Google Bomb).

Worth a look, at least.

Mitch Ratcliffe writes about Swivel, a Web 2.0 site that combines YouTube with Microsoft Excel. Well, not exactly, but sort of.

It is an interesting idea, where you …

1. Upload your dataset to Swivel.
2. Share your data with other users.
3. Chart your data, making it available on the web.
4. Compare your data with the datasets from other users.

Interested? Here is a reasonable example and a not-so-reasonable example of what Swivel can do.

It was a little bid odd to me, at first. Why would I want to provide my data to a web site so that I could look at it the same way that I can on my own computer? Well, I can think of a couple of reasons …

Some Analysis

First, you can share your data with anyone who wants it. This sharing takes two parts, display and anaysis. You can display your data on the web, for everyone to see. Second, you can combine your data with the data that other people have uploaded and perhaps learn something that you didn’t already know.

Now, sharing might be good, and it might not be, depending on your point of view. An enterprise might want to keep it’s information secret, and that makes sense. However, an interesting thing about most data is that it is subject to the network effect, big time. Two unconnected data points might mean something, but you can’t really be sure. However, a hundred data points indicate a trend or a correlation.

Second, you can use a community to learn more about your own data, as well as your correlations. At the bottom of each page, there is a rating that gives a group evaluation of how related and comments about how useful the comparison actually is.

Overall

A site like this could be useful, especially if there could be a concerted effort to provide good information, such as the census, demographic, and geographic information from the US government. It is still in its very rough stages, and it hard to say how much value will come out of combining disparate data sets. Also, I security, information quality, and analysis will be a huge issue here. Still, an interesting idea.