I see the current state of the Internet as the ultimate success … You can deploy a wildly successful Web 2.0 application that serves millions of users and never know how a router, switch or load-balancer works. Even network security and firewalls that were making headline news not more than a few years ago are considered perfunctory. The success of these networking devices and technologies has enabled them to become part of the technology landscape that exists for all to use as they see fit, similar to the microprocessor or electricity.

It is always odd to see the once-glamorous jobs of your youth thrown onto the scrap heap of history (think about the differences in perception between the masons of the middle ages and your local bricklayer). Network Engineers were once the masters of a difficult and arcane field, literally bringing information from chaos. Now, the wizards have been trapped in tiny control panels for now, until they can be embedded in silicon for all time.

This has really got me to thinking about my own field, and its future. What specialties are going to dissapear if data becomes as reliable as electricity? For one thing, I think we would see ETL and Business Analysis become a single career path that is much more abstract and tools-based. With the advent of good BPM, I could see a lot of the scheduling and other mechanics pushed off towards the DBA’s and Systems Administrators. Also, I think that a lot of the hardware could be appliance based, or outsourced completely. Of course, this leaves a great opportunity for open source BI and for nimble players to attack the market and take advantage of the innovators dilemma.

A brave (an infinitely more useful) new world!

The hype about EC2 is that it enables people to ‘rent-a-cloud’, paying for and using as many or as few servers as they wish at any given time.  The problem is that currently these servers are limited in ways that seem minor until you start working with them on practical matters.  This is really visible when working with data and databases on EC2, essentially you have to either change either your tools or your paradigm.  The problem then becomes one of economics, where you try to balance the savings from renting a server and storage against the cost of solving a problem with tools that require a lot of customization.
I would love to be a part of the WeoCEO beta, and see how things work along with S3Infinidisk.  If the problems of addresssing and persistence are solved, then we truly have the ability to scale easily without large-scale customization.  Taking care of these is a giant leap forward, and will help EC2 truly live up to the hype that it has generated in the developer community.

Recently, ran across DBPedia, an organization that is turning Wikipedia entries into RDF, the language used for the semantic web. DBPedia has actually has downloadable datasets based on Wikipedia that are available today. These are datasets that can be queried with existing tools and linked to other datasets. Wow!

Even if you aren’t a Wikipedia fan, this is really a big step forward for the enterprise. Think about the amount of knowledge that exists in your organization that isn’t captured, but is critical to your operations. It has always been a big pain to try and sit down and do formal knowledge engineering. However, most people are comfortable enough with a Wiki to sit down and start typing. For a small organization this might not be such a big deal, but for a larger enterprise this could provide some very useful information.

The first time your Director or CXO can make a financial decision based exclusively on the information from your company wiki, it will have proved its worth. Until then, it is just another trendy tool. The work that DBPedia is doing is an important step in making this a reality.

Most interesting to me was their grid service, which provides an on-demand capability for web hosting that allows a site to handle the slashdot effect without having to blink an eye. Sites (and their corresponding media and applications) are running on multiple servers, which allows traffic to be spread out seamlessly, allowing for spikes in service and usage. This is all done without significant additional configuration, which makes it all the more sweet.

Now, they have had some problems, especially with non-grid oriented applications. However, I think that these are pretty minor compared to the utility that high-performance sites will get from using a grid environment.

Absolutely worth a look …

Basically, when you are using EC2, everything on your server that isn’t statically defined before the machine runs goes bye-bye as soon as the machine reboots. Not a real problem for software, but for data this is a major bummer. Amazon makes S3 storage available without data transfer fees, which is wonderful. However, it takes real effort to transfer data back and forth between the systems (with something like jsh3ll), and most data-centric tools (especially database servers) expect real-time access to a working file system.

S3Infinidisk bridges this gap, allowing an EC2 instance to use S3 like a real Linux filesystem. While it is a bit of hack, it allows data tools to work the way they need to and it allows an EC2 instance to take full advantage of the AWS environment. This is a huge step forward in making EC2 a more usable environment for utility computing!

I haven’t had a chance to try the product out yet, but am excited to do so. I appreciate the licensing structure (free single-user version + commercial high-performance version), although I would prefer seeing open source. Also, since the tool is based on the FUSE subsystem, I could easily see this spreading like wildfire.